[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:34 dist-test-slave-q2kh krb5kdc[29989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:34 dist-test-slave-q2kh krb5kdc[29989](info): set up 2 sockets
Sep 29 18:52:34 dist-test-slave-q2kh krb5kdc[29989](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 18:52:35.819713 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:35 dist-test-slave-q2kh krb5kdc[29989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:35.945377 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35419
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:36.032634 30002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:36.032889 30002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:36.032927 30002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:36.034207 30002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:36.034277 30002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:36.034300 30002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:36.034328 30002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:36.034358 30002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:36.035871 30002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:36.036149 30002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:36.036410 30002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:36.041483 30007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:36.042783 30002 server_base.cc:1034] running on GCE node
W20260929 18:52:36.042943 30010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:36.060154 30008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:36.060614 30002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:36.061069 30002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:36.062263 30002 hybrid_clock.cc:648] HybridClock initialized: now 1790707956062235 us; error 45 us; skew 500 ppm
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:36.064950 30002 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:36.065531 30002 webserver.cc:460] Webserver started at http://127.29.71.190:43703/ using document root <none> and password file <none>
I20260929 18:52:36.065793 30002 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:36.065865 30002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:36.066038 30002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:36.067263 30002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "a34ada525cbd413badaeec97ee23da3f"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "8d40b15c052fdb7ace9a58af75c21c5a"
server_key_iv: "86dc40a740fdcd9b0ae32df105372e79"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.067657 30002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "a34ada525cbd413badaeec97ee23da3f"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "8d40b15c052fdb7ace9a58af75c21c5a"
server_key_iv: "86dc40a740fdcd9b0ae32df105372e79"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.069103 30002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:36.069928 30017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.070130 30002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:36.070281 30002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "a34ada525cbd413badaeec97ee23da3f"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "8d40b15c052fdb7ace9a58af75c21c5a"
server_key_iv: "86dc40a740fdcd9b0ae32df105372e79"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.070356 30002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:36.088227 30002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:36.090821 30002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:36.090934 30002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:36.094913 30002 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:35419
I20260929 18:52:36.094951 30069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:35419 every 8 connection(s)
I20260929 18:52:36.095340 30002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:36.096220 30070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:36.098546 30070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f: Bootstrap starting.
I20260929 18:52:36.099117 30070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:36.099406 30070 log.cc:824] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:36.100217 30070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f: No bootstrap required, opened a new log
I20260929 18:52:36.100648 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30002
I20260929 18:52:36.100778 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:36.101429 29982 external_mini_cluster.cc:1428] Setting key a76a9b762f05f150e4b072855fe83670
I20260929 18:52:36.101971 30070 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34ada525cbd413badaeec97ee23da3f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35419 } }
I20260929 18:52:36.102134 30070 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:36.102171 30070 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34ada525cbd413badaeec97ee23da3f, State: Initialized, Role: FOLLOWER
I20260929 18:52:36.102396 30070 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34ada525cbd413badaeec97ee23da3f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35419 } }
I20260929 18:52:36.102523 30070 raft_consensus.cc:388] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:36.102564 30070 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:36.102607 30070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:36.103461 30070 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34ada525cbd413badaeec97ee23da3f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35419 } }
I20260929 18:52:36.103554 30070 leader_election.cc:302] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a34ada525cbd413badaeec97ee23da3f; no voters:
I20260929 18:52:36.103729 30070 leader_election.cc:288] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:36.103752 30075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:36.103860 30075 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [term 1 LEADER]: Becoming Leader. State: Replica: a34ada525cbd413badaeec97ee23da3f, State: Running, Role: LEADER
I20260929 18:52:36.103940 30075 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34ada525cbd413badaeec97ee23da3f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35419 } }
I20260929 18:52:36.104012 30070 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:36.104260 30075 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a34ada525cbd413badaeec97ee23da3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34ada525cbd413badaeec97ee23da3f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35419 } } }
I20260929 18:52:36.104328 30075 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:36.104444 30075 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34ada525cbd413badaeec97ee23da3f. Latest consensus state: current_term: 1 leader_uuid: "a34ada525cbd413badaeec97ee23da3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a34ada525cbd413badaeec97ee23da3f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35419 } } }
I20260929 18:52:36.104507 30075 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:36.104632 30079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:36.105165 30079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:36.107411 30079 catalog_manager.cc:1347] Generated new cluster ID: 34ff585dd376410dbb20071547285193
I20260929 18:52:36.107465 30079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:36.117395 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.102328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.29.71.190:35419)
0929 18:52:36.102650 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:36.102659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:36.102798 (+ 139us) server_negotiation.cc:407] Connection header received
0929 18:52:36.104952 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:36.104960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:36.105019 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:36.105130 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:36.107053 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.108103 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.109187 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.109625 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.109750 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.113189 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:36.113199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:36.113199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:36.113210 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:36.115404 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.115889 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.115895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.115895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.115988 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.116257 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.116258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.116259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.116529 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:36.116710 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:36.116993 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 18:52:36.117126 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260929 18:52:36.124296 30079 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:36.124994 30079 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:36.128219 30079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a34ada525cbd413badaeec97ee23da3f: Generated new TSK 0
I20260929 18:52:36.128381 30079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:52:36.158272 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35419
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:36.241361 30096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:36.241562 30096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:36.241585 30096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:36.242923 30096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:36.242964 30096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:36.242995 30096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:36.244472 30096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:36.244727 30096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:36.244982 30096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:36.250204 30102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:36.250240 30104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:36.250278 30096 server_base.cc:1034] running on GCE node
W20260929 18:52:36.250241 30101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:36.250643 30096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:36.250892 30096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:36.252056 30096 hybrid_clock.cc:648] HybridClock initialized: now 1790707956252038 us; error 28 us; skew 500 ppm
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:36.254776 30096 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:36.255317 30096 webserver.cc:460] Webserver started at http://127.29.71.129:37631/ using document root <none> and password file <none>
I20260929 18:52:36.255568 30096 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:36.255613 30096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:36.255779 30096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:36.256863 30096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "fd6a0d2eca024077b06eb7321d763d62"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "d34e5485dc1d075c2c2510d5e691ea86"
server_key_iv: "8551fb6203053b1038fa762e334d2a51"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.257221 30096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "fd6a0d2eca024077b06eb7321d763d62"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "d34e5485dc1d075c2c2510d5e691ea86"
server_key_iv: "8551fb6203053b1038fa762e334d2a51"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.258608 30096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:36.259363 30111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.259549 30096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:36.259621 30096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "fd6a0d2eca024077b06eb7321d763d62"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "d34e5485dc1d075c2c2510d5e691ea86"
server_key_iv: "8551fb6203053b1038fa762e334d2a51"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.259714 30096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:36.277536 30096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:36.279752 30096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:36.279995 30096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:36.280253 30096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:36.280582 30096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:36.280615 30096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.280639 30096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:36.280689 30096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.286787 30096 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37163
I20260929 18:52:36.286830 30224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37163 every 8 connection(s)
I20260929 18:52:36.287460 30096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:52:36.293412 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30096
I20260929 18:52:36.293493 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:36.293653 29982 external_mini_cluster.cc:1428] Setting key f9647eaff6372d76060f3affccbbc0ac
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:36.297626 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.288273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:39861 (local address 127.29.71.190:35419)
0929 18:52:36.288389 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:36.288391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:36.288554 (+ 163us) server_negotiation.cc:407] Connection header received
0929 18:52:36.289111 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:36.289112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:36.289124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:36.289172 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:36.290188 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.290723 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.291789 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.292011 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.292098 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.294437 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:36.294438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:36.294439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:36.294445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:36.296138 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.296574 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.296575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.296575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.296626 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.296965 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.296965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.296966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.297072 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:36.297124 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:36.297486 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:52:36.297562 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:52:36.298010 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.288179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35419 (local address 127.29.71.129:39861)
0929 18:52:36.288411 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:36.288462 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:36.288995 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:36.289258 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:36.289262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:36.289871 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:36.290078 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:36.290083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.290854 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.290855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:36.291680 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:36.291682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.292111 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.292112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:36.292245 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.292731 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:36.292739 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:36.294331 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:36.296232 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:36.296238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:36.296240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:36.296482 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:36.296737 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:36.296738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:36.296739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:36.296898 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:36.297141 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:36.297147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:36.297329 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 18:52:36.297452 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:52:36.297636 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260929 18:52:36.298560 30225 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35419
I20260929 18:52:36.298655 30225 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:36.298844 30225 heartbeater.cc:511] Master 127.29.71.190:35419 requested a full tablet report, sending...
I20260929 18:52:36.299342 30034 ts_manager.cc:194] Registered new tserver with Master: fd6a0d2eca024077b06eb7321d763d62 (127.29.71.129:37163)
I20260929 18:52:36.300001 30034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:39861
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:36.322153 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35419
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:36.403896 30230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:36.404150 30230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:36.404191 30230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:36.405951 30230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:36.406019 30230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:36.406066 30230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:36.407539 30230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:36.407831 30230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:36.408071 30230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:36.413339 30235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:36.413429 30230 server_base.cc:1034] running on GCE node
W20260929 18:52:36.413348 30236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:36.413349 30238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:36.413760 30230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:36.414036 30230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:36.415177 30230 hybrid_clock.cc:648] HybridClock initialized: now 1790707956415159 us; error 32 us; skew 500 ppm
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:36.417711 30230 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:36.418277 30230 webserver.cc:460] Webserver started at http://127.29.71.130:39611/ using document root <none> and password file <none>
I20260929 18:52:36.418522 30230 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:36.418591 30230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:36.418725 30230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:36.419788 30230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "f00c0919a7cf49d5815aaab659fc32a3"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "33d97ef855fc69be225945e171d63acc"
server_key_iv: "e496ed6621f89e8e99280bea3499dff0"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.420168 30230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "f00c0919a7cf49d5815aaab659fc32a3"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "33d97ef855fc69be225945e171d63acc"
server_key_iv: "e496ed6621f89e8e99280bea3499dff0"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.421523 30230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:36.422401 30245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.422607 30230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:36.422715 30230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "f00c0919a7cf49d5815aaab659fc32a3"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "33d97ef855fc69be225945e171d63acc"
server_key_iv: "e496ed6621f89e8e99280bea3499dff0"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.422799 30230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:36.434806 30230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:36.437218 30230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:36.437387 30230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:36.437621 30230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:36.437958 30230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:36.438014 30230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.438055 30230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:36.438083 30230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.444450 30230 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:39017
I20260929 18:52:36.444491 30358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:39017 every 8 connection(s)
I20260929 18:52:36.444815 30230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:52:36.446961 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30230
I20260929 18:52:36.447070 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:36.447247 29982 external_mini_cluster.cc:1428] Setting key 19f354d27fd6439408736fcb5bfc10e6
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
I20260929 18:52:36.455873 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.445624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:43445 (local address 127.29.71.190:35419)
0929 18:52:36.445715 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:36.445717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:36.445913 (+ 196us) server_negotiation.cc:407] Connection header received
0929 18:52:36.446474 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:36.446476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:36.446489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:36.446558 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:36.447880 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.448397 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.449762 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.450087 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.450187 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.452779 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:36.452780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:36.452781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:36.452789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:36.454434 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.454840 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.454841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.454841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.454911 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.455169 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.455170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.455170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.455305 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:36.455376 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:36.455655 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 18:52:36.455740 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:52:36.455891 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.445587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35419 (local address 127.29.71.130:43445)
0929 18:52:36.445759 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:36.445794 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:36.446370 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:36.446624 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:36.446627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:36.447469 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:36.447735 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:36.447740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.448491 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.448493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:36.449675 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:36.449677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.450219 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.450220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:36.450326 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.450863 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:36.450871 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:36.452685 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:36.454542 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:36.454548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:36.454550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:36.454760 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:36.454988 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:36.454988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:36.454989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:36.455096 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:36.455371 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:36.455376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:36.455482 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:52:36.455562 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:52:36.455729 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260929 18:52:36.456408 30359 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35419
I20260929 18:52:36.456490 30359 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:36.456677 30359 heartbeater.cc:511] Master 127.29.71.190:35419 requested a full tablet report, sending...
I20260929 18:52:36.457023 30034 ts_manager.cc:194] Registered new tserver with Master: f00c0919a7cf49d5815aaab659fc32a3 (127.29.71.130:39017)
I20260929 18:52:36.457434 30034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:43445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:36.473217 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35419
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:36.558974 30364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:36.559218 30364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:36.559244 30364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:36.560537 30364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:36.560591 30364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:36.560619 30364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:36.562350 30364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:36.562608 30364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:36.562851 30364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:36.568194 30370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:36.568240 30369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:36.568228 30372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:36.568277 30364 server_base.cc:1034] running on GCE node
I20260929 18:52:36.568672 30364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:36.568984 30364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:36.570246 30364 hybrid_clock.cc:648] HybridClock initialized: now 1790707956570228 us; error 26 us; skew 500 ppm
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:36.573091 30364 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:36.573614 30364 webserver.cc:460] Webserver started at http://127.29.71.131:36947/ using document root <none> and password file <none>
I20260929 18:52:36.573873 30364 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:36.573942 30364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:36.574095 30364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:36.575205 30364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "c0ddffc70b534896bdf32a7bf28f7828"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "2096d9f94b7a7bcef3cb652eddace359"
server_key_iv: "f632c2dbd703cd66a552d2709f4a0b20"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.575568 30364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "c0ddffc70b534896bdf32a7bf28f7828"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "2096d9f94b7a7bcef3cb652eddace359"
server_key_iv: "f632c2dbd703cd66a552d2709f4a0b20"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.577366 30364 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 18:52:36.578259 30379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.578451 30364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:36.578557 30364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "c0ddffc70b534896bdf32a7bf28f7828"
format_stamp: "Formatted at 2026-09-29 18:52:36 on dist-test-slave-q2kh"
server_key: "2096d9f94b7a7bcef3cb652eddace359"
server_key_iv: "f632c2dbd703cd66a552d2709f4a0b20"
server_key_version: "encryptionkey@0"
I20260929 18:52:36.578639 30364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:36.601380 30364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:36.603703 30364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:36.603891 30364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:36.604152 30364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:36.604504 30364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:36.604555 30364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.604583 30364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:36.604597 30364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:36.611624 30364 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:41285
I20260929 18:52:36.611708 30492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:41285 every 8 connection(s)
I20260929 18:52:36.612059 30364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:36.618532 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30364
I20260929 18:52:36.618616 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:36.618782 29982 external_mini_cluster.cc:1428] Setting key 0abcf3d3615051e4d9e14f04f786c973
I20260929 18:52:36.621513 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.612869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:41333 (local address 127.29.71.190:35419)
0929 18:52:36.612940 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:36.612942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:36.613174 (+ 232us) server_negotiation.cc:407] Connection header received
0929 18:52:36.613625 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:36.613626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:36.613637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:36.613695 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:36.614666 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.615210 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.616228 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.616418 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.616475 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.618776 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:36.618778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:36.618778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:36.618785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:36.620297 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.620670 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.620670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.620671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.620721 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.620959 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.620960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.620961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.621085 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:36.621146 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:36.621389 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 18:52:36.621452 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:52:36.621691 30495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.612853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35419 (local address 127.29.71.131:41333)
0929 18:52:36.613029 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:36.613066 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:36.613536 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:36.613749 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:36.613752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:36.614378 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:36.614583 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:36.614589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.615316 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.615317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:36.616139 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:36.616141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.616513 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.616514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:36.616621 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.617147 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:36.617155 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:36.618704 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:36.620369 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:36.620373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:36.620375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:36.620599 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:36.620778 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:36.620778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:36.620779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:36.620868 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:36.621146 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:36.621149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:36.621255 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 18:52:36.621337 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:52:36.621450 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260929 18:52:36.622272 30493 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35419
I20260929 18:52:36.622372 30493 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:36.622555 30493 heartbeater.cc:511] Master 127.29.71.190:35419 requested a full tablet report, sending...
I20260929 18:52:36.622934 30034 ts_manager.cc:194] Registered new tserver with Master: c0ddffc70b534896bdf32a7bf28f7828 (127.29.71.131:41285)
I20260929 18:52:36.623318 30034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:41333
I20260929 18:52:36.631122 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:52:36.654992 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.646897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.29.71.129:37163)
0929 18:52:36.647233 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:36.647239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:36.647247 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:52:36.647273 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:36.647275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:36.647288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:36.647347 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:36.647946 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.648673 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.649440 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.649760 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.649816 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.652189 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:36.652198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:36.652198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:36.652205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:36.653693 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.654105 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.654108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.654108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.654231 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.654426 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.654427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.654427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.654571 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:36.654682 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:36.654805 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 18:52:36.654875 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":105,"threads_started":1}
W20260929 18:52:36.655445 30139 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54780
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[29989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:52:36.680014 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:36.672100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.29.71.129:37163)
0929 18:52:36.672186 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:36.672190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:36.672271 (+ 81us) server_negotiation.cc:407] Connection header received
0929 18:52:36.672371 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:36.672372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:36.672386 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:36.672469 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:36.673074 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.673589 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.674437 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:36.674647 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:36.674728 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:36.677087 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:36.677088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:36.677088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:36.677096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:36.678713 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.679136 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.679137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.679138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.679200 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:36.679456 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:36.679457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:36.679457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:36.679569 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:36.679707 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:36.679817 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 18:52:36.679900 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:52:36.680579 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30096
I20260929 18:52:36.685590 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30230
I20260929 18:52:36.691248 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30364
I20260929 18:52:36.695710 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30002
2026-09-29T18:52:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1940 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[30507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[30507](info): set up 2 sockets
Sep 29 18:52:36 dist-test-slave-q2kh krb5kdc[30507](info): commencing operation
krb5kdc: starting...
W20260929 18:52:37.798236 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:37 dist-test-slave-q2kh krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:37.917699 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:39751
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:37.997768 30520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:37.997963 30520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:37.997984 30520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:37.999076 30520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:37.999119 30520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:37.999133 30520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:37.999145 30520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:37.999157 30520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:38.000543 30520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:38.000725 30520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:38.000895 30520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:38.005875 30525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:38.007992 30526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:38.008193 30528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:38.008193 30520 server_base.cc:1034] running on GCE node
I20260929 18:52:38.008507 30520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:38.008814 30520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:38.009986 30520 hybrid_clock.cc:648] HybridClock initialized: now 1790707958009962 us; error 34 us; skew 500 ppm
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:38.012926 30520 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:38.013423 30520 webserver.cc:460] Webserver started at http://127.29.71.190:33705/ using document root <none> and password file <none>
I20260929 18:52:38.013682 30520 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:38.013756 30520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:38.013917 30520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:38.015079 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "2ccf823d61fc4eb2b766b0028da030d3"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "3fd0edcc85895bc624db0f760df947ea"
server_key_iv: "62919308abf699f7fd733b1366783db8"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.015523 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "2ccf823d61fc4eb2b766b0028da030d3"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "3fd0edcc85895bc624db0f760df947ea"
server_key_iv: "62919308abf699f7fd733b1366783db8"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.016906 30520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:38.017668 30535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.017906 30520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:38.018026 30520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "2ccf823d61fc4eb2b766b0028da030d3"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "3fd0edcc85895bc624db0f760df947ea"
server_key_iv: "62919308abf699f7fd733b1366783db8"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.018112 30520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:38.028350 30520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:38.030339 30520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:38.030525 30520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:38.034404 30520 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:39751
I20260929 18:52:38.034436 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:39751 every 8 connection(s)
I20260929 18:52:38.034828 30520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:38.035655 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:38.037871 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3: Bootstrap starting.
I20260929 18:52:38.038503 30588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:38.038738 30588 log.cc:824] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:38.039410 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3: No bootstrap required, opened a new log
I20260929 18:52:38.040935 30588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccf823d61fc4eb2b766b0028da030d3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39751 } }
I20260929 18:52:38.041078 30588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:38.041102 30588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccf823d61fc4eb2b766b0028da030d3, State: Initialized, Role: FOLLOWER
I20260929 18:52:38.041239 30588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccf823d61fc4eb2b766b0028da030d3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39751 } }
I20260929 18:52:38.041320 30588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:38.041344 30588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:38.041373 30588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:38.041905 30588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccf823d61fc4eb2b766b0028da030d3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39751 } }
I20260929 18:52:38.041993 30588 leader_election.cc:302] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ccf823d61fc4eb2b766b0028da030d3; no voters:
I20260929 18:52:38.042109 30588 leader_election.cc:288] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:38.042246 30591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:38.042428 30588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:38.042569 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30520
I20260929 18:52:38.042683 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:38.042840 29982 external_mini_cluster.cc:1428] Setting key 15fac7e6afa371ec0ef1255c27d36dc0
I20260929 18:52:38.043087 30591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2ccf823d61fc4eb2b766b0028da030d3, State: Running, Role: LEADER
I20260929 18:52:38.043197 30591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccf823d61fc4eb2b766b0028da030d3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39751 } }
I20260929 18:52:38.043792 30593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ccf823d61fc4eb2b766b0028da030d3. Latest consensus state: current_term: 1 leader_uuid: "2ccf823d61fc4eb2b766b0028da030d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccf823d61fc4eb2b766b0028da030d3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39751 } } }
I20260929 18:52:38.043893 30593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:38.044116 30602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:38.044195 30592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ccf823d61fc4eb2b766b0028da030d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccf823d61fc4eb2b766b0028da030d3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39751 } } }
I20260929 18:52:38.044303 30592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:38.044787 30602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:38.046872 30610 mvcc.cc:205] Tried to move back new op lower bound from 7334739796155203584 to 7334739796145348608. Current Snapshot: MvccSnapshot[applied={T|T < 7334739796155203584}]
I20260929 18:52:38.047288 30602 catalog_manager.cc:1347] Generated new cluster ID: 508dc29a16ea47379d0704e241e87707
I20260929 18:52:38.047339 30602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:38.056434 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.043528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.29.71.190:39751)
0929 18:52:38.043705 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.043715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.043747 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:52:38.043861 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.043868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.043956 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.044871 (+ 915us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.044967 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.046258 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.047465 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.048158 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.048273 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.051476 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.051481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.051481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.051493 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.054272 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.054704 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.054707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.054708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.054846 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.055077 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.055077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.055078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.055401 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.055973 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.056051 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.056196 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260929 18:52:38.057605 30602 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:38.058085 30602 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:38.060210 30602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ccf823d61fc4eb2b766b0028da030d3: Generated new TSK 0
I20260929 18:52:38.060375 30602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:52:38.089239 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39751
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:38.178128 30614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:38.178344 30614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:38.178367 30614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:38.179526 30614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:38.179574 30614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:38.179600 30614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:38.181126 30614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:38.181363 30614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:38.181612 30614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:38.186713 30620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:38.186710 30619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:38.186829 30614 server_base.cc:1034] running on GCE node
W20260929 18:52:38.186710 30622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:38.187193 30614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:38.187490 30614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:38.188653 30614 hybrid_clock.cc:648] HybridClock initialized: now 1790707958188640 us; error 24 us; skew 500 ppm
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:38.191370 30614 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:38.191877 30614 webserver.cc:460] Webserver started at http://127.29.71.129:43875/ using document root <none> and password file <none>
I20260929 18:52:38.192133 30614 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:38.192205 30614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:38.192364 30614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:38.193413 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "e79e6a8e0b8848bfa6588916b563323e"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "ea6902743a018dc3315c729734a09300"
server_key_iv: "ddbd46134ae1ceb1fc9d05fce1ee56fb"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.193794 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "e79e6a8e0b8848bfa6588916b563323e"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "ea6902743a018dc3315c729734a09300"
server_key_iv: "ddbd46134ae1ceb1fc9d05fce1ee56fb"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.195281 30614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:38.196043 30629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.196234 30614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:38.196358 30614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "e79e6a8e0b8848bfa6588916b563323e"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "ea6902743a018dc3315c729734a09300"
server_key_iv: "ddbd46134ae1ceb1fc9d05fce1ee56fb"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.196437 30614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:38.223308 30614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:38.225347 30614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:38.225523 30614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:38.225795 30614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:38.226126 30614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:38.226212 30614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.226279 30614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:38.226308 30614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.232491 30614 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:46393
I20260929 18:52:38.232539 30742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:46393 every 8 connection(s)
I20260929 18:52:38.232878 30614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:52:38.234352 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30614
I20260929 18:52:38.234433 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:38.234597 29982 external_mini_cluster.cc:1428] Setting key c043285e102ba7e91b7658bd1e8ab92a
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
I20260929 18:52:38.243271 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.233681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:46631 (local address 127.29.71.190:39751)
0929 18:52:38.233758 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.233760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.234007 (+ 247us) server_negotiation.cc:407] Connection header received
0929 18:52:38.235095 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.235096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.235112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.235156 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.235806 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.236382 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.237414 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.237751 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.237827 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.240164 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.240166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.240166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.240174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.241811 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.242271 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.242272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.242273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.242332 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.242587 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.242589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.242589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.242737 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.242809 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.243127 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.243207 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:52:38.243433 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.233655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:39751 (local address 127.29.71.129:46631)
0929 18:52:38.233840 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.233888 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.234976 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.235226 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.235230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.235511 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:38.235700 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.235704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.236481 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.236482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.237319 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.237321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.237869 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.237870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.237993 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.238640 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:38.238649 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:38.240058 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:38.241908 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:38.241914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:38.241916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:38.242140 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:38.242402 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:38.242403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:38.242403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:38.242505 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:38.242810 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:38.242821 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:38.242941 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.243062 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.243229 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":114,"threads_started":1}
I20260929 18:52:38.244035 30743 heartbeater.cc:347] Connected to a master server at 127.29.71.190:39751
I20260929 18:52:38.244153 30743 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:38.244338 30743 heartbeater.cc:511] Master 127.29.71.190:39751 requested a full tablet report, sending...
I20260929 18:52:38.244834 30552 ts_manager.cc:194] Registered new tserver with Master: e79e6a8e0b8848bfa6588916b563323e (127.29.71.129:46393)
I20260929 18:52:38.245510 30552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:46631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:38.262074 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39751
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:38.343717 30748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:38.343962 30748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:38.343999 30748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:38.345275 30748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:38.345353 30748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:38.345400 30748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:38.347039 30748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:38.347309 30748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:38.347540 30748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:38.352624 30753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:38.352622 30756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:38.352671 30748 server_base.cc:1034] running on GCE node
W20260929 18:52:38.352620 30754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:38.353050 30748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:38.353266 30748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:38.354401 30748 hybrid_clock.cc:648] HybridClock initialized: now 1790707958354386 us; error 33 us; skew 500 ppm
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:38.356854 30748 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:38.357283 30748 webserver.cc:460] Webserver started at http://127.29.71.130:40061/ using document root <none> and password file <none>
I20260929 18:52:38.357465 30748 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:38.357507 30748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:38.357595 30748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:38.358639 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "ea83f8b41ac949049a05bb07338d2f12"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "9b34f9eb63d3a33df2ee48e77c8fbfd1"
server_key_iv: "a7ae71c4da14969cdf70372c41b97d47"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.358934 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "ea83f8b41ac949049a05bb07338d2f12"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "9b34f9eb63d3a33df2ee48e77c8fbfd1"
server_key_iv: "a7ae71c4da14969cdf70372c41b97d47"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.360179 30748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:38.360853 30763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.361057 30748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:38.361147 30748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "ea83f8b41ac949049a05bb07338d2f12"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "9b34f9eb63d3a33df2ee48e77c8fbfd1"
server_key_iv: "a7ae71c4da14969cdf70372c41b97d47"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.361222 30748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:38.378487 30748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:38.380887 30748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:38.381067 30748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:38.381309 30748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:38.381649 30748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:38.381708 30748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.381752 30748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:38.381780 30748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.387781 30748 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:46675
I20260929 18:52:38.387822 30876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:46675 every 8 connection(s)
I20260929 18:52:38.388150 30748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:38.397152 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30748
I20260929 18:52:38.397277 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:38.397516 29982 external_mini_cluster.cc:1428] Setting key b11ed3c149f98917d8c462cd56a595fb
I20260929 18:52:38.398032 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.388990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:49323 (local address 127.29.71.190:39751)
0929 18:52:38.389066 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.389068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.389243 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:52:38.390149 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.390151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.390218 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.390280 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.390904 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.391418 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.392383 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.392566 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.392627 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.395012 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.395013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.395014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.395021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.396385 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.396833 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.396833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.396834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.396885 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.397290 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.397291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.397291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.397426 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.397487 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.397847 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.397941 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:52:38.398276 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.388960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:39751 (local address 127.29.71.130:49323)
0929 18:52:38.389141 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.389177 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.390050 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.390359 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.390362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.390614 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:38.390799 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.390803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.391519 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.391520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.392309 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.392311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.392648 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.392649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.392749 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.393304 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:38.393312 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:38.394886 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:38.396480 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:38.396486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:38.396489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:38.396720 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:38.397038 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:38.397039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:38.397039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:38.397184 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:38.397506 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:38.397513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:38.397682 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.397811 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.397978 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260929 18:52:38.398845 30877 heartbeater.cc:347] Connected to a master server at 127.29.71.190:39751
I20260929 18:52:38.398967 30877 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:38.399151 30877 heartbeater.cc:511] Master 127.29.71.190:39751 requested a full tablet report, sending...
I20260929 18:52:38.399587 30552 ts_manager.cc:194] Registered new tserver with Master: ea83f8b41ac949049a05bb07338d2f12 (127.29.71.130:46675)
I20260929 18:52:38.400012 30552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:49323
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:38.425230 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39751
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:38.506357 30882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:38.506629 30882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:38.506672 30882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:38.507882 30882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:38.507966 30882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:38.508013 30882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:38.509588 30882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:38.509883 30882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:38.510141 30882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:38.515401 30890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:38.515470 30888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:38.515586 30887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:38.515626 30882 server_base.cc:1034] running on GCE node
I20260929 18:52:38.515897 30882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:38.516196 30882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:38.517392 30882 hybrid_clock.cc:648] HybridClock initialized: now 1790707958517370 us; error 35 us; skew 500 ppm
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:38.520900 30882 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:38.521581 30882 webserver.cc:460] Webserver started at http://127.29.71.131:42347/ using document root <none> and password file <none>
I20260929 18:52:38.521837 30882 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:38.521884 30882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:38.522043 30882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:38.523150 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "10ff81da24074d1997f812f32ec406a3"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "32d8bd4d075d6a713c28d9685957cb3d"
server_key_iv: "e2a3d45846c4f6d9d4d451a4770204b6"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.523528 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "10ff81da24074d1997f812f32ec406a3"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "32d8bd4d075d6a713c28d9685957cb3d"
server_key_iv: "e2a3d45846c4f6d9d4d451a4770204b6"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.524835 30882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:38.525656 30897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.525867 30882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:38.525939 30882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "10ff81da24074d1997f812f32ec406a3"
format_stamp: "Formatted at 2026-09-29 18:52:38 on dist-test-slave-q2kh"
server_key: "32d8bd4d075d6a713c28d9685957cb3d"
server_key_iv: "e2a3d45846c4f6d9d4d451a4770204b6"
server_key_version: "encryptionkey@0"
I20260929 18:52:38.526036 30882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:38.536276 30882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:38.538421 30882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:38.538589 30882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:38.538828 30882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:38.539153 30882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:38.539186 30882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.539242 30882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:38.539263 30882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:38.545264 30882 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:36177
I20260929 18:52:38.545311 31010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:36177 every 8 connection(s)
I20260929 18:52:38.545620 30882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:52:38.550480 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 30882
I20260929 18:52:38.550598 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:38.550827 29982 external_mini_cluster.cc:1428] Setting key 18f297672d77405b1602f342737de117
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:38.555362 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.546426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:45479 (local address 127.29.71.190:39751)
0929 18:52:38.546473 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.546475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.546645 (+ 170us) server_negotiation.cc:407] Connection header received
0929 18:52:38.547666 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.547667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.547679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.547730 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.548348 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.548850 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.549892 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.550068 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.550120 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.552609 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.552610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.552610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.552616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.554035 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.554507 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.554508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.554508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.554557 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.554771 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.554771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.554772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.554870 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.554921 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.555238 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.555300 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 18:52:38.555692 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.546414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:39751 (local address 127.29.71.131:45479)
0929 18:52:38.546541 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.546577 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.547553 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.547795 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.547798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.548071 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:38.548263 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.548267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.548945 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.548946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.549813 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.549816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.550143 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.550144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.550297 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.550873 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:38.550884 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:38.552521 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:38.554117 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:38.554123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:38.554125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:38.554428 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:38.554621 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:38.554621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:38.554622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:38.554715 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:38.554958 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:38.554964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:38.555102 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.555220 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.555382 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260929 18:52:38.556296 31011 heartbeater.cc:347] Connected to a master server at 127.29.71.190:39751
I20260929 18:52:38.556394 31011 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:38.556572 31011 heartbeater.cc:511] Master 127.29.71.190:39751 requested a full tablet report, sending...
I20260929 18:52:38.556990 30552 ts_manager.cc:194] Registered new tserver with Master: 10ff81da24074d1997f812f32ec406a3 (127.29.71.131:36177)
I20260929 18:52:38.557369 30552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:45479
I20260929 18:52:38.562886 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:52:38.570974 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.564514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.29.71.190:39751)
0929 18:52:38.564595 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.564598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.564687 (+ 89us) server_negotiation.cc:407] Connection header received
0929 18:52:38.564788 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.564790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.564800 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.564873 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.565471 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.565949 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.566768 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.566959 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.567040 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.568268 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.568270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.568270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.568277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.569700 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.570098 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.570099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.570099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.570201 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.570410 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.570411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.570411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.570571 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.570676 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.570788 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.570870 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:52:38.573345 30552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:52:38.574980 30552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:38.581069 31022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.577251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:36177 (local address 127.0.0.1:49314)
0929 18:52:38.577395 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.577426 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.577483 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.577749 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.577752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.577754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:38.577828 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.577830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.579054 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.579055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.579923 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.579925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.580768 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.580769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.580846 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.580863 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.580930 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.580994 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260929 18:52:38.581468 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.577377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.29.71.131:36177)
0929 18:52:38.577592 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.577596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.577605 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:38.577647 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.577648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.577660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.577746 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:38.577888 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.578957 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.580012 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.580696 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.580752 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.580792 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.581355 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.581395 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":123,"threads_started":1}
I20260929 18:52:38.581833 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.578031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.29.71.130:46675)
0929 18:52:38.578275 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.578280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.578290 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:38.578328 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.578330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.578345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.578570 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:38.578654 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.579735 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.580902 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.581463 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.581550 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.581602 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.581702 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.581745 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260929 18:52:38.582007 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.577527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46675 (local address 127.0.0.1:50930)
0929 18:52:38.577643 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.577653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.577712 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.578445 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.578446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.578448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:38.578495 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.578496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.579853 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.579854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.580814 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.580816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.581563 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.581564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.581629 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.581632 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.581883 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.581939 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20260929 18:52:38.582405 30944 tablet_service.cc:1467] Processing CreateTablet for tablet 89ac421f0b1a499fa8e9feccf288c627 (DEFAULT_TABLE table=test-table [id=c3e3061a92e84d9e8cc9dd52c47cd0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:38.582775 30944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ac421f0b1a499fa8e9feccf288c627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:38.582703 30811 tablet_service.cc:1467] Processing CreateTablet for tablet 89ac421f0b1a499fa8e9feccf288c627 (DEFAULT_TABLE table=test-table [id=c3e3061a92e84d9e8cc9dd52c47cd0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:38.583096 30811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ac421f0b1a499fa8e9feccf288c627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:38.583214 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.577734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:46393 (local address 127.0.0.1:49482)
0929 18:52:38.577832 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.577841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.577879 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.578532 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.578532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.578535 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:38.578580 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.578582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.580923 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.580924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.581961 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.581964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.582957 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.582958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.583037 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.583041 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.583104 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.583151 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":66,"threads_started":1}
I20260929 18:52:38.583655 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.578104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.29.71.129:46393)
0929 18:52:38.578290 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.578294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.578304 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:38.578343 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.578345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.578360 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.578433 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:38.578666 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.579740 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.582062 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.582782 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.582869 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.582922 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.583137 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.583579 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260929 18:52:38.584473 30677 tablet_service.cc:1467] Processing CreateTablet for tablet 89ac421f0b1a499fa8e9feccf288c627 (DEFAULT_TABLE table=test-table [id=c3e3061a92e84d9e8cc9dd52c47cd0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:38.584806 30677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ac421f0b1a499fa8e9feccf288c627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:38.585490 31028 tablet_bootstrap.cc:492] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12: Bootstrap starting.
I20260929 18:52:38.586201 31028 tablet_bootstrap.cc:654] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:38.586565 31029 tablet_bootstrap.cc:492] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3: Bootstrap starting.
I20260929 18:52:38.586809 31028 log.cc:824] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:38.587224 31029 tablet_bootstrap.cc:654] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:38.587610 31028 tablet_bootstrap.cc:492] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12: No bootstrap required, opened a new log
I20260929 18:52:38.587684 31028 ts_tablet_manager.cc:1397] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:52:38.587716 31029 log.cc:824] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:38.588413 31029 tablet_bootstrap.cc:492] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3: No bootstrap required, opened a new log
I20260929 18:52:38.588560 31032 tablet_bootstrap.cc:492] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e: Bootstrap starting.
I20260929 18:52:38.588483 31029 ts_tablet_manager.cc:1397] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:52:38.589217 31032 tablet_bootstrap.cc:654] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:38.589553 31028 raft_consensus.cc:348] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.589720 31028 raft_consensus.cc:374] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:38.589762 31028 raft_consensus.cc:729] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea83f8b41ac949049a05bb07338d2f12, State: Initialized, Role: FOLLOWER
I20260929 18:52:38.589758 31032 log.cc:824] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:38.589906 31028 consensus_queue.cc:260] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.590256 31028 ts_tablet_manager.cc:1428] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:38.590294 30877 heartbeater.cc:503] Master 127.29.71.190:39751 was elected leader, sending a full tablet report...
I20260929 18:52:38.591266 31032 tablet_bootstrap.cc:492] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e: No bootstrap required, opened a new log
I20260929 18:52:38.591336 31032 ts_tablet_manager.cc:1397] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:52:38.591566 31029 raft_consensus.cc:348] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.591739 31029 raft_consensus.cc:374] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:38.591773 31029 raft_consensus.cc:729] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ff81da24074d1997f812f32ec406a3, State: Initialized, Role: FOLLOWER
I20260929 18:52:38.591886 31029 consensus_queue.cc:260] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.592254 31029 ts_tablet_manager.cc:1428] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:38.592320 31011 heartbeater.cc:503] Master 127.29.71.190:39751 was elected leader, sending a full tablet report...
I20260929 18:52:38.593372 31032 raft_consensus.cc:348] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.593497 31032 raft_consensus.cc:374] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:38.593516 31032 raft_consensus.cc:729] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79e6a8e0b8848bfa6588916b563323e, State: Initialized, Role: FOLLOWER
I20260929 18:52:38.593626 31032 consensus_queue.cc:260] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.593852 31032 ts_tablet_manager.cc:1428] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:52:38.593896 30743 heartbeater.cc:503] Master 127.29.71.190:39751 was elected leader, sending a full tablet report...
I20260929 18:52:38.635198 31036 raft_consensus.cc:482] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:38.635368 31036 raft_consensus.cc:504] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.635754 31036 leader_election.cc:288] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10ff81da24074d1997f812f32ec406a3 (127.29.71.131:36177), ea83f8b41ac949049a05bb07338d2f12 (127.29.71.130:46675)
W20260929 18:52:38.638751 30878 tablet.cc:2369] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:52:38.639662 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.636144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:51019 (local address 127.29.71.131:36177)
0929 18:52:38.636196 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.636199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.636214 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:52:38.636248 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.636249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.636260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.636329 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:38.636533 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.637315 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.638809 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.639283 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.639358 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.639366 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.639556 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.639586 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:52:38.639683 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.636020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:36177 (local address 127.29.71.129:51019)
0929 18:52:38.636093 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.636107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.636188 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.636346 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.636347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.636350 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:38.636429 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.636432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.637402 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.637403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.638702 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.638707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.639387 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.639388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.639477 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.639481 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.639540 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.639587 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":2}
I20260929 18:52:38.639799 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.636161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46675 (local address 127.29.71.129:52993)
0929 18:52:38.636381 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:38.636392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:38.636454 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:38.636618 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:38.636619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:38.636620 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:38.636669 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.636671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.637534 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.637534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.638892 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:38.638894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.639544 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.639545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:38.639606 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.639609 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:38.639658 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:52:38.639687 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":205,"thread_start_us":146,"threads_started":1}
I20260929 18:52:38.639875 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.636244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:52993 (local address 127.29.71.130:46675)
0929 18:52:38.636287 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.636289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.636487 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:52:38.636537 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.636539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.636548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.636607 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:38.636742 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.637463 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.638980 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.639418 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.639466 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.639474 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.639688 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.639736 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:52:38.640151 30831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ac421f0b1a499fa8e9feccf288c627" candidate_uuid: "e79e6a8e0b8848bfa6588916b563323e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea83f8b41ac949049a05bb07338d2f12" is_pre_election: true
I20260929 18:52:38.640162 30965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ac421f0b1a499fa8e9feccf288c627" candidate_uuid: "e79e6a8e0b8848bfa6588916b563323e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ff81da24074d1997f812f32ec406a3" is_pre_election: true
I20260929 18:52:38.640276 30831 raft_consensus.cc:2454] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79e6a8e0b8848bfa6588916b563323e in term 0.
I20260929 18:52:38.640317 30965 raft_consensus.cc:2454] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79e6a8e0b8848bfa6588916b563323e in term 0.
I20260929 18:52:38.640547 30630 leader_election.cc:302] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10ff81da24074d1997f812f32ec406a3, e79e6a8e0b8848bfa6588916b563323e; no voters:
I20260929 18:52:38.640682 31036 raft_consensus.cc:2781] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:38.640739 31036 raft_consensus.cc:482] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:38.640764 31036 raft_consensus.cc:3037] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:38.641453 31036 raft_consensus.cc:504] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.641594 31036 leader_election.cc:288] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [CANDIDATE]: Term 1 election: Requested vote from peers 10ff81da24074d1997f812f32ec406a3 (127.29.71.131:36177), ea83f8b41ac949049a05bb07338d2f12 (127.29.71.130:46675)
I20260929 18:52:38.641758 30965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ac421f0b1a499fa8e9feccf288c627" candidate_uuid: "e79e6a8e0b8848bfa6588916b563323e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10ff81da24074d1997f812f32ec406a3"
I20260929 18:52:38.641768 30831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ac421f0b1a499fa8e9feccf288c627" candidate_uuid: "e79e6a8e0b8848bfa6588916b563323e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea83f8b41ac949049a05bb07338d2f12"
I20260929 18:52:38.641837 30831 raft_consensus.cc:3037] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:38.641860 30965 raft_consensus.cc:3037] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:38.642606 30831 raft_consensus.cc:2454] T 89ac421f0b1a499fa8e9feccf288c627 P ea83f8b41ac949049a05bb07338d2f12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79e6a8e0b8848bfa6588916b563323e in term 1.
I20260929 18:52:38.642742 30965 raft_consensus.cc:2454] T 89ac421f0b1a499fa8e9feccf288c627 P 10ff81da24074d1997f812f32ec406a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79e6a8e0b8848bfa6588916b563323e in term 1.
I20260929 18:52:38.642807 30630 leader_election.cc:302] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e79e6a8e0b8848bfa6588916b563323e, ea83f8b41ac949049a05bb07338d2f12; no voters:
I20260929 18:52:38.642926 31036 raft_consensus.cc:2781] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:38.643155 31036 raft_consensus.cc:686] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [term 1 LEADER]: Becoming Leader. State: Replica: e79e6a8e0b8848bfa6588916b563323e, State: Running, Role: LEADER
I20260929 18:52:38.643281 31036 consensus_queue.cc:237] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } }
I20260929 18:52:38.644212 30552 catalog_manager.cc:5495] T 89ac421f0b1a499fa8e9feccf288c627 P e79e6a8e0b8848bfa6588916b563323e reported cstate change: term changed from 0 to 1, leader changed from <none> to e79e6a8e0b8848bfa6588916b563323e (127.29.71.129). New cstate: current_term: 1 leader_uuid: "e79e6a8e0b8848bfa6588916b563323e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10ff81da24074d1997f812f32ec406a3" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea83f8b41ac949049a05bb07338d2f12" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79e6a8e0b8848bfa6588916b563323e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 46393 } health_report { overall_health: HEALTHY } } }
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:52:38.662382 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.655062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.29.71.129:46393)
0929 18:52:38.655106 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.655108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.655181 (+ 73us) server_negotiation.cc:407] Connection header received
0929 18:52:38.655236 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.655236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.655244 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.655296 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.655856 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.656334 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.657050 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.657240 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.657297 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.659766 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.659777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.659777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.659784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.661318 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.661634 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.661637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.661638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.661694 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.661872 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.661873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.661873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.662019 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.662152 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.662254 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.662317 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:52:38.686416 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.678699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.29.71.129:46393)
0929 18:52:38.678772 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.678774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.678862 (+ 88us) server_negotiation.cc:407] Connection header received
0929 18:52:38.678957 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.678958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.678977 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.679047 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.679656 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.680159 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.680967 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.681172 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.681264 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.683724 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.683725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.683725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.683732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.685175 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.685550 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.685551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.685552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.685608 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.685795 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.685795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.685796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.685927 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.686009 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.686271 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.686348 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[30507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:52:38.709898 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:38.702381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.29.71.129:46393)
0929 18:52:38.702510 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:38.702513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:38.702628 (+ 115us) server_negotiation.cc:407] Connection header received
0929 18:52:38.702714 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:38.702716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:38.702730 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:38.702812 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:38.703494 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.704023 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.704879 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:38.705089 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:38.705209 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:38.707691 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:38.707693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:38.707693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:38.707700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:38.708971 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.709286 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.709287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.709288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.709339 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:38.709510 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:38.709511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:38.709511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:38.709610 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:38.709687 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:38.709773 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 18:52:38.709830 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 18:52:38.710466 30657 tablet_service.cc:2908] Rejecting scan request for tablet 89ac421f0b1a499fa8e9feccf288c627: Uninitialized: clean time has not yet been initialized
I20260929 18:52:38.711628 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30614
I20260929 18:52:38.716428 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30748
I20260929 18:52:38.721194 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30882
I20260929 18:52:38.725677 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 30520
2026-09-29T18:52:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2030 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[31052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[31052](info): set up 2 sockets
Sep 29 18:52:38 dist-test-slave-q2kh krb5kdc[31052](info): commencing operation
krb5kdc: starting...
W20260929 18:52:39.778046 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:39 dist-test-slave-q2kh krb5kdc[31052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:39.909739 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:36815
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:36815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:39.991206 31065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:39.991461 31065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:39.991501 31065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:39.992655 31065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:39.992748 31065 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:52:39.992769 31065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:39.992791 31065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:39.992820 31065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:39.992836 31065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:39.994788 31065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:36815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:36815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:39.995054 31065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:39.995316 31065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:40.000211 31071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:40.000211 31073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:40.000280 31070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:40.000542 31065 server_base.cc:1034] running on GCE node
I20260929 18:52:40.000813 31065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:40.001119 31065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:40.002287 31065 hybrid_clock.cc:648] HybridClock initialized: now 1790707960002282 us; error 38 us; skew 500 ppm
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:40.013585 31065 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:40.014115 31065 webserver.cc:460] Webserver started at http://127.29.71.190:39939/ using document root <none> and password file <none>
I20260929 18:52:40.014411 31065 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:40.014497 31065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:40.014663 31065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:40.015823 31065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "c4cd0cfdea4a45fa82fbad0a9170c814"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "5355c3652be2121083514995bdf3441d"
server_key_iv: "3656ed69f5e9dab78effb03f34be4b3f"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.016209 31065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "c4cd0cfdea4a45fa82fbad0a9170c814"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "5355c3652be2121083514995bdf3441d"
server_key_iv: "3656ed69f5e9dab78effb03f34be4b3f"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.017647 31065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:40.018497 31080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.018718 31065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:40.018810 31065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "c4cd0cfdea4a45fa82fbad0a9170c814"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "5355c3652be2121083514995bdf3441d"
server_key_iv: "3656ed69f5e9dab78effb03f34be4b3f"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.018867 31065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:40.064275 31065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:40.066767 31065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:40.066884 31065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:40.071317 31065 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:36815
I20260929 18:52:40.071411 31142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:36815 every 8 connection(s)
I20260929 18:52:40.071759 31065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:40.072510 31143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:40.075042 31143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: Bootstrap starting.
I20260929 18:52:40.075394 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 31065
I20260929 18:52:40.075531 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:40.075703 29982 external_mini_cluster.cc:1428] Setting key 797fe94f01c8383aa97b63bf97d96e37
I20260929 18:52:40.075937 31143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:40.076340 31143 log.cc:824] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:40.077349 31143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: No bootstrap required, opened a new log
I20260929 18:52:40.079190 31143 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36815 } }
I20260929 18:52:40.079337 31143 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:40.079363 31143 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cd0cfdea4a45fa82fbad0a9170c814, State: Initialized, Role: FOLLOWER
I20260929 18:52:40.079514 31143 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36815 } }
I20260929 18:52:40.079599 31143 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:40.079649 31143 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:40.079708 31143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:40.080291 31143 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36815 } }
I20260929 18:52:40.080425 31143 leader_election.cc:302] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4cd0cfdea4a45fa82fbad0a9170c814; no voters:
I20260929 18:52:40.080632 31143 leader_election.cc:288] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:40.080662 31148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:40.080797 31148 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [term 1 LEADER]: Becoming Leader. State: Replica: c4cd0cfdea4a45fa82fbad0a9170c814, State: Running, Role: LEADER
I20260929 18:52:40.080888 31148 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36815 } }
I20260929 18:52:40.080919 31143 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:40.081245 31148 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4cd0cfdea4a45fa82fbad0a9170c814. Latest consensus state: current_term: 1 leader_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36815 } } }
I20260929 18:52:40.081351 31148 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:40.081259 31149 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cd0cfdea4a45fa82fbad0a9170c814" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36815 } } }
I20260929 18:52:40.081506 31149 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814 [sys.catalog]: This master's current role is: LEADER
W20260929 18:52:40.082967 31163 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:52:40.083057 31163 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:52:40.083163 31152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:40.083371 31152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:40.086591 31152 catalog_manager.cc:1347] Generated new cluster ID: 2a07d2eccb4342448efeed9fa7cec5f0
I20260929 18:52:40.086668 31152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:52:40.088330 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.076291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.29.71.190:36815)
0929 18:52:40.076525 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.076534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.076560 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:52:40.077290 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.077300 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.077386 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.077609 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.078281 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.079684 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.080461 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.081267 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.081735 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.084401 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.084405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.084406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.084417 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.086472 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.086899 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.086902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.086903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.087008 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.087256 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.087257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.087257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.087577 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.087742 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.087843 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.088095 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260929 18:52:40.092772 31152 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:40.093293 31152 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:40.097620 31152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: Generated new TSK 0
I20260929 18:52:40.097805 31152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:52:40.129436 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36815
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:40.217149 31169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:40.217402 31169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:40.217445 31169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:40.218729 31169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:40.218799 31169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:40.218835 31169 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:52:40.218874 31169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:40.220355 31169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:36815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:40.220672 31169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:40.220916 31169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:40.226261 31177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:40.226315 31174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:40.226271 31175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:40.226723 31169 server_base.cc:1034] running on GCE node
I20260929 18:52:40.226938 31169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:40.227183 31169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:40.228338 31169 hybrid_clock.cc:648] HybridClock initialized: now 1790707960228322 us; error 27 us; skew 500 ppm
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:40.230927 31169 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:40.231397 31169 webserver.cc:460] Webserver started at http://127.29.71.129:36311/ using document root <none> and password file <none>
I20260929 18:52:40.231647 31169 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:40.231720 31169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:40.231875 31169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:40.232954 31169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "ac261e49d28340f6b55836e9667562f3"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "6b8fe096aa8a359d9c9fd4138eff8cef"
server_key_iv: "dcbd180d6b10d2860614b162dc44ad91"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.233345 31169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "ac261e49d28340f6b55836e9667562f3"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "6b8fe096aa8a359d9c9fd4138eff8cef"
server_key_iv: "dcbd180d6b10d2860614b162dc44ad91"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.234831 31169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:40.235621 31184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.235877 31169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:40.235980 31169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "ac261e49d28340f6b55836e9667562f3"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "6b8fe096aa8a359d9c9fd4138eff8cef"
server_key_iv: "dcbd180d6b10d2860614b162dc44ad91"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.236083 31169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:40.257448 31169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:40.259814 31169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:40.260002 31169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:40.260526 31169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:40.260591 31169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.260630 31169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:40.260663 31169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.267606 31169 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:44235
I20260929 18:52:40.267710 31299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:44235 every 8 connection(s)
I20260929 18:52:40.268050 31169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:40.272207 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.261279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:42507 (local address 127.29.71.190:36815)
0929 18:52:40.261347 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.261350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.261543 (+ 193us) server_negotiation.cc:407] Connection header received
0929 18:52:40.262292 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.262293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.262308 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.262386 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.263292 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.264154 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.265789 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.266190 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.266269 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.269219 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.269220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.269220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.269227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.270947 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.271344 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.271345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.271345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.271397 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.271631 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.271631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.271631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.271734 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.271783 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.272060 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.272139 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:52:40.272356 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.261228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36815 (local address 127.29.71.129:42507)
0929 18:52:40.261416 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.261455 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.262116 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.262475 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.262480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.262878 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:40.263177 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.263183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.264267 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.264268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.265697 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.265700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.266321 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.266322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.266485 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.267145 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:40.267153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:40.269067 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:40.271030 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.271037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.271038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.271261 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.271460 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.271461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.271461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.271571 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.271791 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:40.271795 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:40.271916 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.272006 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.272134 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260929 18:52:40.273051 31300 heartbeater.cc:347] Connected to a master server at 127.29.71.190:36815
I20260929 18:52:40.273134 31300 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:40.273322 31300 heartbeater.cc:511] Master 127.29.71.190:36815 requested a full tablet report, sending...
I20260929 18:52:40.273897 31097 ts_manager.cc:194] Registered new tserver with Master: ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235)
I20260929 18:52:40.274860 31097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:42507
I20260929 18:52:40.275403 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 31169
I20260929 18:52:40.275532 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:40.275698 29982 external_mini_cluster.cc:1428] Setting key 41a5cabc80a01fb7b6b5fe39a4d5a6c5
I20260929 18:52:40.280650 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.274308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:59575 (local address 127.29.71.190:36815)
0929 18:52:40.274349 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.274351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.274444 (+ 93us) server_negotiation.cc:407] Connection header received
0929 18:52:40.274507 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.274508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.274517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.274573 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.274974 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.275799 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.276713 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.277082 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.277169 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.278182 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.278189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.278189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.278196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.279548 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.279902 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.279904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.279904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.279961 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.280165 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.280166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.280166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.280273 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.280351 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.280491 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.280561 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:52:40.280669 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.274254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36815 (local address 127.29.71.129:59575)
0929 18:52:40.274391 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.274401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.274453 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.274594 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.274595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.274796 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:40.274853 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.274855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.275916 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.275917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.276616 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.276619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.277194 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.277195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.277259 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.277764 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:40.277768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:40.278045 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:40.279646 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.279647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.279647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.279816 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.280030 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.280031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.280032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.280089 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.280343 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:40.280344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:40.280389 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.280458 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.280563 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:40.304803 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36815
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:40.387221 31309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:40.387430 31309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:40.387454 31309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:40.388566 31309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:40.388597 31309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:40.388616 31309 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:52:40.388633 31309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:40.390093 31309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:36815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:40.390377 31309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:40.390568 31309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:40.395704 31317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:40.395783 31314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:40.395701 31315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:40.395927 31309 server_base.cc:1034] running on GCE node
I20260929 18:52:40.396165 31309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:40.396414 31309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:40.397583 31309 hybrid_clock.cc:648] HybridClock initialized: now 1790707960397582 us; error 43 us; skew 500 ppm
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:40.400271 31309 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:40.400730 31309 webserver.cc:460] Webserver started at http://127.29.71.130:42445/ using document root <none> and password file <none>
I20260929 18:52:40.400936 31309 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:40.400975 31309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:40.401074 31309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:40.402079 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "83586e15ca04260d39cb21853fb280f3"
server_key_iv: "bb6a8a48f1401521d07315ccd0832a8b"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.402451 31309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "83586e15ca04260d39cb21853fb280f3"
server_key_iv: "bb6a8a48f1401521d07315ccd0832a8b"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.403852 31309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:40.404628 31324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.404838 31309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:40.404911 31309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "83586e15ca04260d39cb21853fb280f3"
server_key_iv: "bb6a8a48f1401521d07315ccd0832a8b"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.405045 31309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:40.418990 31309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:40.421428 31309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:40.421579 31309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:40.422271 31309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:40.422493 31309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.422525 31309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:40.422595 31309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:40.429956 31309 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:44515
I20260929 18:52:40.430040 31439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:44515 every 8 connection(s)
I20260929 18:52:40.430393 31309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:52:40.432276 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.422652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:45427 (local address 127.29.71.190:36815)
0929 18:52:40.422698 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.422700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.423047 (+ 347us) server_negotiation.cc:407] Connection header received
0929 18:52:40.423528 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.423529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.423543 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.423622 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.424280 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.425126 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.426188 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.426544 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.426621 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.429147 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.429148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.429149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.429156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.430866 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.431289 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.431290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.431290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.431348 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.431609 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.431611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.431611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.431747 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.431840 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.432107 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.432184 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:52:40.432477 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.422684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36815 (local address 127.29.71.130:45427)
0929 18:52:40.422894 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.422935 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.423412 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.423696 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.423700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.423979 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:40.424160 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.424165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.425239 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.425240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.426054 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.426056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.426640 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.426641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.426782 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.427351 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:40.427362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:40.429032 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:40.430967 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.430972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.430974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.431193 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.431426 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.431426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.431427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.431537 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.431835 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:40.431841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:40.431976 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.432089 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.432242 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260929 18:52:40.432989 31440 heartbeater.cc:347] Connected to a master server at 127.29.71.190:36815
I20260929 18:52:40.433060 31440 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:40.433254 31440 heartbeater.cc:511] Master 127.29.71.190:36815 requested a full tablet report, sending...
I20260929 18:52:40.433646 31097 ts_manager.cc:194] Registered new tserver with Master: 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130:44515)
I20260929 18:52:40.434031 31097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:45427
I20260929 18:52:40.439499 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.434137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:59017 (local address 127.29.71.190:36815)
0929 18:52:40.434192 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.434194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.434330 (+ 136us) server_negotiation.cc:407] Connection header received
0929 18:52:40.434371 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.434372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.434404 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.434464 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.434745 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.435247 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.435975 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.436148 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.436201 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.437011 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.437012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.437012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.437018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.438437 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.438844 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.438845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.438845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.438895 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.439072 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.439072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.439072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.439202 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.439255 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.439377 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.439430 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:52:40.439505 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.434068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36815 (local address 127.29.71.130:59017)
0929 18:52:40.434253 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.434264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.434321 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.434471 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.434472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.434614 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:40.434671 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.434672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.435342 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.435343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.435900 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.435902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.436217 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.436218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.436272 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.436640 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:40.436644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:40.436933 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:40.438512 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.438513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.438513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.438686 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.438957 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.438958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.438958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.439014 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.439259 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:40.439260 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:40.439296 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.439361 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.439434 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260929 18:52:40.440029 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 31309
I20260929 18:52:40.440104 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:40.440232 29982 external_mini_cluster.cc:1428] Setting key a972443fe02e0c2713e10baf1598aad9
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:40.465955 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36815
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:40.548903 31449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:40.549109 31449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:40.549132 31449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:40.550370 31449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:40.550411 31449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:40.550433 31449 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:52:40.550496 31449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:40.551913 31449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:36815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:40.552158 31449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:40.552397 31449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:40.557516 31457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:40.557542 31455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:40.557555 31449 server_base.cc:1034] running on GCE node
W20260929 18:52:40.557520 31454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:40.557926 31449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:40.558138 31449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:40.559379 31449 hybrid_clock.cc:648] HybridClock initialized: now 1790707960559369 us; error 31 us; skew 500 ppm
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:40.561980 31449 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:40.562556 31449 webserver.cc:460] Webserver started at http://127.29.71.131:36413/ using document root <none> and password file <none>
I20260929 18:52:40.562810 31449 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:40.562891 31449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:40.563042 31449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:40.564082 31449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "582b5de4e89e43b28bc5e2acb3139423"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "df12fc71c55b600b454b682c6b4646cb"
server_key_iv: "4988e4dcfb36d376a1225f88906d1ab2"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.564460 31449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "582b5de4e89e43b28bc5e2acb3139423"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "df12fc71c55b600b454b682c6b4646cb"
server_key_iv: "4988e4dcfb36d376a1225f88906d1ab2"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.565812 31449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:40.566753 31464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.566927 31449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:52:40.567018 31449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "582b5de4e89e43b28bc5e2acb3139423"
format_stamp: "Formatted at 2026-09-29 18:52:40 on dist-test-slave-q2kh"
server_key: "df12fc71c55b600b454b682c6b4646cb"
server_key_iv: "4988e4dcfb36d376a1225f88906d1ab2"
server_key_version: "encryptionkey@0"
I20260929 18:52:40.567082 31449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:40.582297 31449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:40.584337 31449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:40.584498 31449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:40.585084 31449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:40.585124 31449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.585146 31449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:40.585160 31449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:40.592901 31449 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:35265
I20260929 18:52:40.592974 31579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:35265 every 8 connection(s)
I20260929 18:52:40.593365 31449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:52:40.595430 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.585555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:45661 (local address 127.29.71.190:36815)
0929 18:52:40.585633 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.585636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.585811 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:52:40.586368 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.586370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.586384 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.586461 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.587046 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.587909 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.589409 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.589686 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.589778 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.592519 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.592520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.592521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.592527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.594139 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.594613 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.594613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.594614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.594667 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.594847 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.594848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.594848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.594954 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.595022 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.595301 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.595365 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:52:40.595593 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.585532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36815 (local address 127.29.71.131:45661)
0929 18:52:40.585671 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.585712 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.586260 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.586556 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.586559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.586793 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:40.586970 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.586974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.588029 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.588031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.589307 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.589309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.589786 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.589787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.589921 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.590487 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:40.590496 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:40.592402 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:40.594290 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.594296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.594298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.594547 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.594710 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.594711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.594712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.594796 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.595037 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:40.595042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:40.595172 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.595266 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.595390 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260929 18:52:40.596096 31580 heartbeater.cc:347] Connected to a master server at 127.29.71.190:36815
I20260929 18:52:40.596176 31580 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:40.596370 31580 heartbeater.cc:511] Master 127.29.71.190:36815 requested a full tablet report, sending...
I20260929 18:52:40.596810 31096 ts_manager.cc:194] Registered new tserver with Master: 582b5de4e89e43b28bc5e2acb3139423 (127.29.71.131:35265)
I20260929 18:52:40.597361 31096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:45661
I20260929 18:52:40.600592 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 31449
I20260929 18:52:40.600684 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:40.600869 29982 external_mini_cluster.cc:1428] Setting key f538d65bef714a216f614206416c6ce1
I20260929 18:52:40.601826 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:52:40.603742 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.597232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36815 (local address 127.29.71.131:57223)
0929 18:52:40.597345 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.597357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.597407 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.597644 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.597661 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.597892 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:40.597982 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.597984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.598927 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.598928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.599457 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.599458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.599804 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.599805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.599896 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.600327 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:40.600332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:40.600764 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:40.602550 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.602552 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.602552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.602779 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.602971 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:40.602972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:40.602972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:40.603039 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:40.603347 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:40.603348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:40.603399 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.603477 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.603598 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260929 18:52:40.603734 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.597288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:57223 (local address 127.29.71.190:36815)
0929 18:52:40.597529 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.597530 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.597539 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:40.597556 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.597556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.597566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.597642 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.598143 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.598737 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.599535 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.599708 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.599759 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.600860 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.600861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.600861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.600867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.602470 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.602861 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.602861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.602862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.602910 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.603140 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.603141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.603141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.603267 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.603340 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.603496 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.603595 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:52:40 dist-test-slave-q2kh krb5kdc[31052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:40.626773 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.619377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.29.71.190:36815)
0929 18:52:40.619448 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.619451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.619459 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:52:40.619490 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.619491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.619500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.619569 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:40.620159 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.620667 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.621512 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.621713 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.621814 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.624238 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:40.624240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:40.624240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:40.624247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:40.625500 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.625827 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.625828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.625828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.625879 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:40.626058 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:40.626058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:40.626058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:40.626383 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:40.626477 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.626632 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.626702 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:52:40.628669 31096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:52:40.630458 31096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:40.637713 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.633288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.29.71.130:44515)
0929 18:52:40.633467 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.633472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.633662 (+ 190us) server_negotiation.cc:407] Connection header received
0929 18:52:40.633802 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.633805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.633823 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.633914 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:40.634185 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.635395 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.636590 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.637315 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.637405 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.637465 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.637566 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.637608 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260929 18:52:40.637732 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.633152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:44515 (local address 127.0.0.1:49864)
0929 18:52:40.633532 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.633576 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.633664 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.633995 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.633998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.634001 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:40.634092 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.634094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.635492 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.635493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.636457 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.636460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.637415 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.637416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.637475 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.637487 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.637550 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.637632 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":23,"thread_start_us":48,"threads_started":1}
I20260929 18:52:40.637929 31601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.633410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:35265 (local address 127.0.0.1:48058)
0929 18:52:40.633776 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.633786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.633861 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.634130 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.634130 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.634133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:40.634220 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.634222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.635482 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.635483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.636715 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.636718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.637604 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.637605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.637656 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.637660 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.637711 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.637747 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":44,"thread_start_us":61,"threads_started":1}
I20260929 18:52:40.637925 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.633592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.29.71.131:35265)
0929 18:52:40.633780 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.633784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.633829 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:52:40.634018 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.634021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.634032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.634117 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:40.634302 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.635386 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.636808 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.637517 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.637584 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.637628 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.637791 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.637836 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260929 18:52:40.638912 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.633152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:44235 (local address 127.0.0.1:36336)
0929 18:52:40.633620 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.633630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.633924 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.634377 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.634377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.634379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:40.634407 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.634408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.635313 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.635314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.636716 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.636718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.638633 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.638633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.638704 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.638708 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.638804 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.638850 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":37,"thread_start_us":86,"threads_started":1}
I20260929 18:52:40.638971 31600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.633330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.29.71.129:44235)
0929 18:52:40.633512 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.633516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.633707 (+ 191us) server_negotiation.cc:407] Connection header received
0929 18:52:40.634221 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.634224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.634240 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.634309 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:40.634493 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.635203 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.637662 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.638490 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.638561 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.638615 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.638820 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.638876 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260929 18:52:40.638990 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 02391506d1bc4848a6e1650536ba68ff (DEFAULT_TABLE table=test-table [id=38f3843d9c9c4af7b5977ac22e9464ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:40.639365 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02391506d1bc4848a6e1650536ba68ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:40.639608 31234 tablet_service.cc:1467] Processing CreateTablet for tablet 02391506d1bc4848a6e1650536ba68ff (DEFAULT_TABLE table=test-table [id=38f3843d9c9c4af7b5977ac22e9464ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:40.639923 31234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02391506d1bc4848a6e1650536ba68ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:40.642060 31603 tablet_bootstrap.cc:492] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Bootstrap starting.
I20260929 18:52:40.643112 31603 tablet_bootstrap.cc:654] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:40.643172 31604 tablet_bootstrap.cc:492] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Bootstrap starting.
I20260929 18:52:40.643268 31374 tablet_service.cc:1467] Processing CreateTablet for tablet 02391506d1bc4848a6e1650536ba68ff (DEFAULT_TABLE table=test-table [id=38f3843d9c9c4af7b5977ac22e9464ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:40.643461 31603 log.cc:824] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:40.643571 31374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02391506d1bc4848a6e1650536ba68ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:40.643786 31604 tablet_bootstrap.cc:654] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:40.644075 31604 log.cc:824] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:40.644305 31603 tablet_bootstrap.cc:492] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: No bootstrap required, opened a new log
I20260929 18:52:40.644364 31603 ts_tablet_manager.cc:1397] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:52:40.644841 31604 tablet_bootstrap.cc:492] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: No bootstrap required, opened a new log
I20260929 18:52:40.644899 31604 ts_tablet_manager.cc:1397] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:52:40.646412 31603 raft_consensus.cc:348] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.646581 31603 raft_consensus.cc:374] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:40.646621 31603 raft_consensus.cc:729] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582b5de4e89e43b28bc5e2acb3139423, State: Initialized, Role: FOLLOWER
I20260929 18:52:40.646739 31603 consensus_queue.cc:260] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.647037 31603 ts_tablet_manager.cc:1428] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 18:52:40.646872 31604 raft_consensus.cc:348] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.647213 31580 heartbeater.cc:503] Master 127.29.71.190:36815 was elected leader, sending a full tablet report...
I20260929 18:52:40.647195 31604 raft_consensus.cc:374] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:40.647266 31604 raft_consensus.cc:729] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac261e49d28340f6b55836e9667562f3, State: Initialized, Role: FOLLOWER
I20260929 18:52:40.647387 31604 consensus_queue.cc:260] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.647768 31604 ts_tablet_manager.cc:1428] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:52:40.647778 31300 heartbeater.cc:503] Master 127.29.71.190:36815 was elected leader, sending a full tablet report...
I20260929 18:52:40.647965 31607 tablet_bootstrap.cc:492] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Bootstrap starting.
I20260929 18:52:40.648605 31607 tablet_bootstrap.cc:654] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:40.648958 31607 log.cc:824] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:40.649785 31607 tablet_bootstrap.cc:492] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: No bootstrap required, opened a new log
I20260929 18:52:40.649873 31607 ts_tablet_manager.cc:1397] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:52:40.651650 31607 raft_consensus.cc:348] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.651794 31607 raft_consensus.cc:374] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:40.651841 31607 raft_consensus.cc:729] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cd6b2b6cc64e03bb3974e2e8e8071c, State: Initialized, Role: FOLLOWER
I20260929 18:52:40.651949 31607 consensus_queue.cc:260] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.652190 31607 ts_tablet_manager.cc:1428] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 18:52:40.652233 31440 heartbeater.cc:503] Master 127.29.71.190:36815 was elected leader, sending a full tablet report...
W20260929 18:52:40.681026 31441 tablet.cc:2369] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:52:40.768828 31301 tablet.cc:2369] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:52:40.844049 31581 tablet.cc:2369] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:52:40.864900 31611 raft_consensus.cc:482] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:40.865060 31611 raft_consensus.cc:504] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.865525 31611 leader_election.cc:288] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235), 582b5de4e89e43b28bc5e2acb3139423 (127.29.71.131:35265)
I20260929 18:52:40.868407 31600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.865851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:38999 (local address 127.29.71.129:44235)
0929 18:52:40.865930 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.865933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.865943 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:40.865972 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.865973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.865982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.866056 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:40.866246 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.866742 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.867670 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.868034 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.868123 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.868132 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.868293 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.868338 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:52:40.868405 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.865737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:44235 (local address 127.29.71.130:38999)
0929 18:52:40.865790 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.865802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.865901 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.866060 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.866060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.866062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:40.866141 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.866143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.866836 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.866837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.867572 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.867574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.868121 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.868122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.868202 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.868206 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.868285 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.868332 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:52:40.868477 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.865850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:57173 (local address 127.29.71.131:35265)
0929 18:52:40.865897 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.865900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.866020 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:52:40.866090 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.866091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.866101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.866178 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:40.866337 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.866801 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.867757 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.868123 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.868211 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.868219 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.868372 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.868408 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:52:40.868525 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.865761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:35265 (local address 127.29.71.130:57173)
0929 18:52:40.865931 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:40.865940 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:40.866033 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:40.866177 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:40.866177 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:40.866178 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:40.866259 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.866265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.866900 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.866901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.867653 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:40.867655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.868211 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.868212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:40.868289 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.868292 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:40.868368 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:52:40.868405 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
I20260929 18:52:40.868775 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02391506d1bc4848a6e1650536ba68ff" candidate_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac261e49d28340f6b55836e9667562f3" is_pre_election: true
I20260929 18:52:40.868831 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02391506d1bc4848a6e1650536ba68ff" candidate_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582b5de4e89e43b28bc5e2acb3139423" is_pre_election: true
I20260929 18:52:40.868969 31534 raft_consensus.cc:2454] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63cd6b2b6cc64e03bb3974e2e8e8071c in term 0.
I20260929 18:52:40.868973 31254 raft_consensus.cc:2454] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63cd6b2b6cc64e03bb3974e2e8e8071c in term 0.
I20260929 18:52:40.869237 31328 leader_election.cc:302] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 582b5de4e89e43b28bc5e2acb3139423, 63cd6b2b6cc64e03bb3974e2e8e8071c; no voters:
I20260929 18:52:40.869369 31611 raft_consensus.cc:2781] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:40.869424 31611 raft_consensus.cc:482] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:40.869449 31611 raft_consensus.cc:3037] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:40.870510 31611 raft_consensus.cc:504] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.870702 31611 leader_election.cc:288] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [CANDIDATE]: Term 1 election: Requested vote from peers ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235), 582b5de4e89e43b28bc5e2acb3139423 (127.29.71.131:35265)
I20260929 18:52:40.870831 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02391506d1bc4848a6e1650536ba68ff" candidate_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac261e49d28340f6b55836e9667562f3"
I20260929 18:52:40.870893 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02391506d1bc4848a6e1650536ba68ff" candidate_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582b5de4e89e43b28bc5e2acb3139423"
I20260929 18:52:40.870939 31254 raft_consensus.cc:3037] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:40.870955 31534 raft_consensus.cc:3037] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:40.871819 31254 raft_consensus.cc:2454] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63cd6b2b6cc64e03bb3974e2e8e8071c in term 1.
I20260929 18:52:40.871830 31534 raft_consensus.cc:2454] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63cd6b2b6cc64e03bb3974e2e8e8071c in term 1.
I20260929 18:52:40.872064 31326 leader_election.cc:302] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63cd6b2b6cc64e03bb3974e2e8e8071c, ac261e49d28340f6b55836e9667562f3; no voters:
I20260929 18:52:40.872229 31611 raft_consensus.cc:2781] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:40.872411 31611 raft_consensus.cc:686] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 LEADER]: Becoming Leader. State: Replica: 63cd6b2b6cc64e03bb3974e2e8e8071c, State: Running, Role: LEADER
I20260929 18:52:40.872527 31611 consensus_queue.cc:237] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.873535 31096 catalog_manager.cc:5495] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c reported cstate change: term changed from 0 to 1, leader changed from <none> to 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130). New cstate: current_term: 1 leader_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } health_report { overall_health: UNKNOWN } } }
I20260929 18:52:40.922319 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:40.919830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.29.71.130:44515)
0929 18:52:40.919924 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:40.919926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:40.919936 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:40.919978 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:40.919978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:40.919996 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:40.920057 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:40.920193 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.920676 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.921477 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:40.921699 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:40.921753 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:40.921943 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:40.922037 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:40.922119 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:40.922181 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 18:52:40.922233 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:52:40.924081 31534 raft_consensus.cc:1261] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Refusing update from remote peer 63cd6b2b6cc64e03bb3974e2e8e8071c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:40.924078 31254 raft_consensus.cc:1261] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Refusing update from remote peer 63cd6b2b6cc64e03bb3974e2e8e8071c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:40.924501 31613 consensus_queue.cc:1035] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:40.924649 31611 consensus_queue.cc:1035] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [LEADER]: Connected to new peer: Peer: permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:40.927269 31617 mvcc.cc:205] Tried to move back new op lower bound from 7334739807942197248 to 7334739807734448128. Current Snapshot: MvccSnapshot[applied={T|T < 7334739807942197248}]
I20260929 18:52:40.927403 31618 mvcc.cc:205] Tried to move back new op lower bound from 7334739807942197248 to 7334739807734448128. Current Snapshot: MvccSnapshot[applied={T|T < 7334739807942197248 or (T in {7334739807942197248})}]
I20260929 18:52:40.930249 31097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:40.930341 31097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:40.931126 31097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: Sending DeleteTablet for 3 replicas of tablet 02391506d1bc4848a6e1650536ba68ff
I20260929 18:52:40.931494 31374 tablet_service.cc:1514] Processing DeleteTablet for tablet 02391506d1bc4848a6e1650536ba68ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:40 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:49864
I20260929 18:52:40.931506 31234 tablet_service.cc:1514] Processing DeleteTablet for tablet 02391506d1bc4848a6e1650536ba68ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:40 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:36336
I20260929 18:52:40.931730 31627 tablet_replica.cc:330] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: stopping tablet replica
I20260929 18:52:40.931746 31626 tablet_replica.cc:330] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: stopping tablet replica
I20260929 18:52:40.931774 31514 tablet_service.cc:1514] Processing DeleteTablet for tablet 02391506d1bc4848a6e1650536ba68ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:40 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:48058
I20260929 18:52:40.931823 31627 raft_consensus.cc:2229] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:40.931838 31626 raft_consensus.cc:2229] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:40.931942 31627 raft_consensus.cc:2258] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:40.931964 31626 raft_consensus.cc:2258] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:40.932443 31627 ts_tablet_manager.cc:1905] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:40.932372 31097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:52:40.932482 31628 tablet_replica.cc:330] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: stopping tablet replica
W20260929 18:52:40.932528 31097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:40.932561 31628 raft_consensus.cc:2229] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:40.932703 31628 raft_consensus.cc:2258] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:40.933122 31628 ts_tablet_manager.cc:1905] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:40.933950 31234 tablet_service.cc:1467] Processing CreateTablet for tablet 135567fe216e4a1d9356ed4d8afb9e23 (DEFAULT_TABLE table=test-table [id=f0cc5035dba1447faf767b5fe61415a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:40.934070 31234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135567fe216e4a1d9356ed4d8afb9e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:40.934427 31374 tablet_service.cc:1467] Processing CreateTablet for tablet 135567fe216e4a1d9356ed4d8afb9e23 (DEFAULT_TABLE table=test-table [id=f0cc5035dba1447faf767b5fe61415a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:40.934530 31374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135567fe216e4a1d9356ed4d8afb9e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:40.934723 31627 ts_tablet_manager.cc:1918] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:52:40.934828 31627 log.cc:1196] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/02391506d1bc4848a6e1650536ba68ff
I20260929 18:52:40.934840 31628 ts_tablet_manager.cc:1918] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:52:40.934906 31628 log.cc:1196] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/02391506d1bc4848a6e1650536ba68ff
I20260929 18:52:40.935145 31628 ts_tablet_manager.cc:1939] T 02391506d1bc4848a6e1650536ba68ff P 582b5de4e89e43b28bc5e2acb3139423: Deleting consensus metadata
I20260929 18:52:40.935338 31627 ts_tablet_manager.cc:1939] T 02391506d1bc4848a6e1650536ba68ff P 63cd6b2b6cc64e03bb3974e2e8e8071c: Deleting consensus metadata
I20260929 18:52:40.935770 31084 catalog_manager.cc:4841] TS 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130:44515): tablet 02391506d1bc4848a6e1650536ba68ff (table test-table [id=38f3843d9c9c4af7b5977ac22e9464ac]) successfully deleted
I20260929 18:52:40.935906 31607 tablet_bootstrap.cc:492] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: Bootstrap starting.
I20260929 18:52:40.935962 31514 tablet_service.cc:1467] Processing CreateTablet for tablet 135567fe216e4a1d9356ed4d8afb9e23 (DEFAULT_TABLE table=test-table [id=f0cc5035dba1447faf767b5fe61415a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:40.936058 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135567fe216e4a1d9356ed4d8afb9e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:40.936301 31607 tablet_bootstrap.cc:654] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:40.936961 31607 tablet_bootstrap.cc:492] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: No bootstrap required, opened a new log
I20260929 18:52:40.937011 31607 ts_tablet_manager.cc:1397] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:40.937155 31607 raft_consensus.cc:348] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.937611 31607 raft_consensus.cc:374] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:40.937633 31607 raft_consensus.cc:729] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cd6b2b6cc64e03bb3974e2e8e8071c, State: Initialized, Role: FOLLOWER
I20260929 18:52:40.937681 31607 consensus_queue.cc:260] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.937784 31607 ts_tablet_manager.cc:1428] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:52:40.937791 31603 tablet_bootstrap.cc:492] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: Bootstrap starting.
I20260929 18:52:40.937870 31084 catalog_manager.cc:4841] TS 582b5de4e89e43b28bc5e2acb3139423 (127.29.71.131:35265): tablet 02391506d1bc4848a6e1650536ba68ff (table test-table [id=38f3843d9c9c4af7b5977ac22e9464ac]) successfully deleted
I20260929 18:52:40.938218 31603 tablet_bootstrap.cc:654] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:40.938824 31603 tablet_bootstrap.cc:492] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: No bootstrap required, opened a new log
I20260929 18:52:40.938879 31603 ts_tablet_manager.cc:1397] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:40.939033 31603 raft_consensus.cc:348] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.939108 31603 raft_consensus.cc:374] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:40.939128 31603 raft_consensus.cc:729] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582b5de4e89e43b28bc5e2acb3139423, State: Initialized, Role: FOLLOWER
I20260929 18:52:40.939172 31603 consensus_queue.cc:260] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.939267 31603 ts_tablet_manager.cc:1428] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:40.939384 31604 tablet_bootstrap.cc:492] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: Bootstrap starting.
I20260929 18:52:40.939770 31604 tablet_bootstrap.cc:654] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:40.940104 31626 ts_tablet_manager.cc:1905] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:40.940430 31604 tablet_bootstrap.cc:492] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: No bootstrap required, opened a new log
I20260929 18:52:40.940476 31604 ts_tablet_manager.cc:1397] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:40.940608 31604 raft_consensus.cc:348] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.940692 31604 raft_consensus.cc:374] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:40.940718 31604 raft_consensus.cc:729] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac261e49d28340f6b55836e9667562f3, State: Initialized, Role: FOLLOWER
I20260929 18:52:40.940760 31604 consensus_queue.cc:260] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:40.940914 31604 ts_tablet_manager.cc:1428] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:40.941769 31626 ts_tablet_manager.cc:1918] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:52:40.941871 31626 log.cc:1196] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/02391506d1bc4848a6e1650536ba68ff
I20260929 18:52:40.942430 31626 ts_tablet_manager.cc:1939] T 02391506d1bc4848a6e1650536ba68ff P ac261e49d28340f6b55836e9667562f3: Deleting consensus metadata
I20260929 18:52:40.942865 31082 catalog_manager.cc:4841] TS ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235): tablet 02391506d1bc4848a6e1650536ba68ff (table test-table [id=38f3843d9c9c4af7b5977ac22e9464ac]) successfully deleted
I20260929 18:52:41.179706 31608 raft_consensus.cc:482] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:41.179801 31608 raft_consensus.cc:504] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.180112 31608 leader_election.cc:288] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235), 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130:44515)
I20260929 18:52:41.183363 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.180301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:44235 (local address 127.29.71.131:45785)
0929 18:52:41.180481 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:41.180503 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:41.180559 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:41.180702 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:41.180703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:41.180706 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:41.180781 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.180783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.181522 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.181523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.182552 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.182554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.183097 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.183098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.183164 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.183168 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:41.183231 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:52:41.183277 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260929 18:52:41.183409 31600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.180371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:45785 (local address 127.29.71.129:44235)
0929 18:52:41.180417 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.180419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.180579 (+ 160us) server_negotiation.cc:407] Connection header received
0929 18:52:41.180614 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.180615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.180626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.180686 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:41.180854 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.181442 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.182645 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.183019 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.183074 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.183081 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.183265 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.183327 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:52:41.183561 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.180362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:44515 (local address 127.29.71.131:47473)
0929 18:52:41.180589 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:41.180601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:41.180668 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:41.180855 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:41.180856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:41.180858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:41.180914 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.180916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.181673 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.181674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.182584 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.182607 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.183295 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.183296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.183358 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.183361 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:41.183419 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:52:41.183458 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":4,"thread_start_us":57,"threads_started":1}
I20260929 18:52:41.183565 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.180414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:47473 (local address 127.29.71.130:44515)
0929 18:52:41.180461 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.180463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.180691 (+ 228us) server_negotiation.cc:407] Connection header received
0929 18:52:41.180730 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.180731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.180743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.180796 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:41.181023 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.181591 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.182710 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.183212 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.183275 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.183282 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.183446 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.183488 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:52:41.183820 31394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135567fe216e4a1d9356ed4d8afb9e23" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" is_pre_election: true
I20260929 18:52:41.183943 31394 raft_consensus.cc:2454] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 0.
I20260929 18:52:41.184010 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135567fe216e4a1d9356ed4d8afb9e23" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac261e49d28340f6b55836e9667562f3" is_pre_election: true
I20260929 18:52:41.184099 31254 raft_consensus.cc:2454] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 0.
I20260929 18:52:41.184141 31468 leader_election.cc:302] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 582b5de4e89e43b28bc5e2acb3139423, 63cd6b2b6cc64e03bb3974e2e8e8071c; no voters:
I20260929 18:52:41.184280 31608 raft_consensus.cc:2781] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:41.184324 31608 raft_consensus.cc:482] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:41.184351 31608 raft_consensus.cc:3037] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:41.184989 31608 raft_consensus.cc:504] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.185140 31608 leader_election.cc:288] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 election: Requested vote from peers ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235), 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130:44515)
I20260929 18:52:41.185308 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135567fe216e4a1d9356ed4d8afb9e23" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac261e49d28340f6b55836e9667562f3"
I20260929 18:52:41.185308 31394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135567fe216e4a1d9356ed4d8afb9e23" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c"
I20260929 18:52:41.185387 31394 raft_consensus.cc:3037] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:41.185387 31254 raft_consensus.cc:3037] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:41.186625 31254 raft_consensus.cc:2454] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 1.
I20260929 18:52:41.186625 31394 raft_consensus.cc:2454] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 1.
I20260929 18:52:41.186807 31468 leader_election.cc:302] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 582b5de4e89e43b28bc5e2acb3139423, 63cd6b2b6cc64e03bb3974e2e8e8071c; no voters:
I20260929 18:52:41.186935 31608 raft_consensus.cc:2781] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:41.187071 31608 raft_consensus.cc:686] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 1 LEADER]: Becoming Leader. State: Replica: 582b5de4e89e43b28bc5e2acb3139423, State: Running, Role: LEADER
I20260929 18:52:41.187151 31608 consensus_queue.cc:237] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.187844 31095 catalog_manager.cc:5495] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 reported cstate change: term changed from 0 to 1, leader changed from <none> to 582b5de4e89e43b28bc5e2acb3139423 (127.29.71.131). New cstate: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } health_report { overall_health: HEALTHY } } }
Sep 29 18:52:41 dist-test-slave-q2kh krb5kdc[31052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:41.254438 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.247078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.29.71.190:36815)
0929 18:52:41.247244 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.247245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.247261 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:52:41.247336 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.247337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.247347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.247412 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:41.248045 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.248566 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.249298 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.249482 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.249533 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.251654 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:41.251656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:41.251656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:41.251662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:41.253181 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:41.253562 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:41.253562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:41.253563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:41.253615 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:41.253886 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:41.253886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:41.253887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:41.254002 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:41.254052 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.254287 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.254364 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260929 18:52:41.254680 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.246980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36815 (local address 127.0.0.1:40466)
0929 18:52:41.247188 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:41.247201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:41.247277 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:41.247458 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:41.247459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:41.247854 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:41.247909 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.247912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.248649 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.248650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.249224 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.249226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.249549 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.249550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.249589 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.250084 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:41.250089 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:41.251550 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:41.253274 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:41.253281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:41.253283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:41.253491 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:41.253697 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:41.253698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:41.253698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:41.253817 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:41.254124 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:41.254130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:41.254206 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:52:41.254268 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 18:52:41.254535 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":72,"thread_start_us":120,"threads_started":1}
I20260929 18:52:41.257165 31095 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:40466:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:52:41.257418 31095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:41.259006 31374 tablet_service.cc:1467] Processing CreateTablet for tablet db534ab7ff9c40f0a4e3e0a8cffef8fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3439b60be90946b0b5ebfc26236f749c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:52:41.258998 31234 tablet_service.cc:1467] Processing CreateTablet for tablet db534ab7ff9c40f0a4e3e0a8cffef8fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3439b60be90946b0b5ebfc26236f749c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:52:41.259022 31514 tablet_service.cc:1467] Processing CreateTablet for tablet db534ab7ff9c40f0a4e3e0a8cffef8fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3439b60be90946b0b5ebfc26236f749c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:52:41.259145 31374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db534ab7ff9c40f0a4e3e0a8cffef8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:41.259150 31514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db534ab7ff9c40f0a4e3e0a8cffef8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:41.259150 31234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db534ab7ff9c40f0a4e3e0a8cffef8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:41.260704 31604 tablet_bootstrap.cc:492] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: Bootstrap starting.
I20260929 18:52:41.260704 31603 tablet_bootstrap.cc:492] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: Bootstrap starting.
I20260929 18:52:41.260707 31607 tablet_bootstrap.cc:492] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: Bootstrap starting.
I20260929 18:52:41.261018 31607 tablet_bootstrap.cc:654] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:41.261147 31604 tablet_bootstrap.cc:654] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:41.261148 31603 tablet_bootstrap.cc:654] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:41.261811 31603 tablet_bootstrap.cc:492] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: No bootstrap required, opened a new log
I20260929 18:52:41.261857 31603 ts_tablet_manager.cc:1397] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:41.261842 31607 tablet_bootstrap.cc:492] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: No bootstrap required, opened a new log
I20260929 18:52:41.261920 31607 ts_tablet_manager.cc:1397] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:41.262023 31603 raft_consensus.cc:348] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.262044 31607 raft_consensus.cc:348] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.262099 31603 raft_consensus.cc:374] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:41.262120 31603 raft_consensus.cc:729] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582b5de4e89e43b28bc5e2acb3139423, State: Initialized, Role: FOLLOWER
I20260929 18:52:41.262122 31607 raft_consensus.cc:374] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:41.262147 31607 raft_consensus.cc:729] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cd6b2b6cc64e03bb3974e2e8e8071c, State: Initialized, Role: FOLLOWER
I20260929 18:52:41.262209 31603 consensus_queue.cc:260] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.262223 31607 consensus_queue.cc:260] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.262315 31603 ts_tablet_manager.cc:1428] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:41.262326 31607 ts_tablet_manager.cc:1428] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:41.262370 31608 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.262454 31608 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:41.262461 31621 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.262529 31604 tablet_bootstrap.cc:492] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: No bootstrap required, opened a new log
I20260929 18:52:41.262540 31621 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:41.262578 31604 ts_tablet_manager.cc:1397] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:52:41.262753 31604 raft_consensus.cc:348] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.262831 31604 raft_consensus.cc:374] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:41.262851 31604 raft_consensus.cc:729] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac261e49d28340f6b55836e9667562f3, State: Initialized, Role: FOLLOWER
I20260929 18:52:41.262889 31604 consensus_queue.cc:260] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.262997 31604 ts_tablet_manager.cc:1428] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:41.263089 31609 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.263196 31609 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:41.302896 31608 raft_consensus.cc:482] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:41.303001 31608 raft_consensus.cc:504] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.303159 31608 leader_election.cc:288] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235), 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130:44515)
I20260929 18:52:41.303395 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db534ab7ff9c40f0a4e3e0a8cffef8fe" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac261e49d28340f6b55836e9667562f3" is_pre_election: true
I20260929 18:52:41.303418 31394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db534ab7ff9c40f0a4e3e0a8cffef8fe" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" is_pre_election: true
I20260929 18:52:41.303506 31254 raft_consensus.cc:2454] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 0.
I20260929 18:52:41.303506 31394 raft_consensus.cc:2454] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 0.
I20260929 18:52:41.303670 31466 leader_election.cc:302] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 582b5de4e89e43b28bc5e2acb3139423, ac261e49d28340f6b55836e9667562f3; no voters:
I20260929 18:52:41.303812 31608 raft_consensus.cc:2781] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:41.303885 31608 raft_consensus.cc:482] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:41.303913 31608 raft_consensus.cc:3037] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:41.304616 31608 raft_consensus.cc:504] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.304773 31608 leader_election.cc:288] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 election: Requested vote from peers ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235), 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130:44515)
I20260929 18:52:41.304957 31394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db534ab7ff9c40f0a4e3e0a8cffef8fe" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c"
I20260929 18:52:41.304960 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db534ab7ff9c40f0a4e3e0a8cffef8fe" candidate_uuid: "582b5de4e89e43b28bc5e2acb3139423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac261e49d28340f6b55836e9667562f3"
I20260929 18:52:41.305039 31394 raft_consensus.cc:3037] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:41.305039 31254 raft_consensus.cc:3037] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:41.305768 31394 raft_consensus.cc:2454] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 1.
I20260929 18:52:41.305769 31254 raft_consensus.cc:2454] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582b5de4e89e43b28bc5e2acb3139423 in term 1.
I20260929 18:52:41.305922 31466 leader_election.cc:302] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 582b5de4e89e43b28bc5e2acb3139423, ac261e49d28340f6b55836e9667562f3; no voters:
I20260929 18:52:41.306039 31608 raft_consensus.cc:2781] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:41.306129 31608 raft_consensus.cc:686] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [term 1 LEADER]: Becoming Leader. State: Replica: 582b5de4e89e43b28bc5e2acb3139423, State: Running, Role: LEADER
I20260929 18:52:41.306196 31608 consensus_queue.cc:237] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } }
I20260929 18:52:41.306519 31634 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: TxnStatusTablet state changed. Reason: New leader 582b5de4e89e43b28bc5e2acb3139423. Latest consensus state: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.306607 31634 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:52:41.306780 31647 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:52:41.306896 31647 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:52:41.306870 31097 catalog_manager.cc:5495] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 reported cstate change: term changed from 0 to 1, leader changed from <none> to 582b5de4e89e43b28bc5e2acb3139423 (127.29.71.131). New cstate: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } health_report { overall_health: HEALTHY } } }
I20260929 18:52:41.313969 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.311087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:35265 (local address 127.0.0.1:48070)
0929 18:52:41.311149 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:41.311210 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:41.311271 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:41.311471 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:41.311471 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:41.311472 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:52:41.311517 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.311519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.312402 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.312403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.312971 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.312972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.313449 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.313450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.313497 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.313511 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:52:41.313797 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:52:41.313803 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:52:41.313872 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:52:41.313914 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 18:52:41.314005 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.311169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.29.71.131:35265)
0929 18:52:41.311209 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.311211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.311252 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:52:41.311360 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.311361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.311379 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.311455 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:41.311582 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.312316 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.313067 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.313361 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.313438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.313607 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:41.313726 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:41.313804 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.313888 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.313909 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:52:41.315168 31254 raft_consensus.cc:1261] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Refusing update from remote peer 582b5de4e89e43b28bc5e2acb3139423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:41.315199 31394 raft_consensus.cc:1261] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Refusing update from remote peer 582b5de4e89e43b28bc5e2acb3139423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:41.315421 31608 consensus_queue.cc:1035] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:41.315567 31608 consensus_queue.cc:1035] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:41.316097 31621 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: TxnStatusTablet state changed. Reason: New leader 582b5de4e89e43b28bc5e2acb3139423. Latest consensus state: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.316191 31621 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:41.316018 31609 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: TxnStatusTablet state changed. Reason: New leader 582b5de4e89e43b28bc5e2acb3139423. Latest consensus state: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.316517 31609 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:41.316708 31634 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.316780 31634 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:52:41.317822 31634 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.317914 31634 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 582b5de4e89e43b28bc5e2acb3139423: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:52:41.318287 31616 mvcc.cc:205] Tried to move back new op lower bound from 7334739809544818688 to 7334739809510502400. Current Snapshot: MvccSnapshot[applied={T|T < 7334739809544818688}]
I20260929 18:52:41.318373 31621 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.318470 31621 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P 63cd6b2b6cc64e03bb3974e2e8e8071c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:41.318805 31609 tablet_replica.cc:439] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "582b5de4e89e43b28bc5e2acb3139423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 } } peers { permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 } } peers { permanent_uuid: "582b5de4e89e43b28bc5e2acb3139423" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 35265 } } }
I20260929 18:52:41.318889 31609 tablet_replica.cc:442] T db534ab7ff9c40f0a4e3e0a8cffef8fe P ac261e49d28340f6b55836e9667562f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:41.321967 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.319343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.29.71.131:35265)
0929 18:52:41.319378 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.319380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.319393 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:52:41.319408 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.319409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.319416 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.319481 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:41.319637 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.320524 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.321283 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.321512 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.321585 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.321749 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:41.321779 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:41.321826 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.321884 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.321900 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 18:52:41.327363 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.324438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:36745 (local address 127.29.71.131:35265)
0929 18:52:41.324472 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.324474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.324611 (+ 137us) server_negotiation.cc:407] Connection header received
0929 18:52:41.324634 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.324635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.324644 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.324706 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:41.324829 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.325569 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.326564 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.326859 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.326932 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.327074 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:41.327105 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:41.327160 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.327245 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.327262 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:52:41.327484 31655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.324374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:35265 (local address 127.29.71.131:36745)
0929 18:52:41.324530 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:41.324541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:41.324600 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:41.324713 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:41.324714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:41.324715 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:52:41.324769 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.324770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.325650 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.325651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.326487 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:41.326489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.326937 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.326938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:41.326985 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.326997 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:52:41.327187 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:52:41.327190 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:41.327241 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:52:41.327278 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260929 18:52:41.329761 31254 raft_consensus.cc:1261] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Refusing update from remote peer 582b5de4e89e43b28bc5e2acb3139423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:41.329761 31394 raft_consensus.cc:1261] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Refusing update from remote peer 582b5de4e89e43b28bc5e2acb3139423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:41.329946 31634 consensus_queue.cc:1035] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac261e49d28340f6b55836e9667562f3" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:41.330039 31608 consensus_queue.cc:1035] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cd6b2b6cc64e03bb3974e2e8e8071c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:41.341699 31096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:41.341804 31096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:41.342445 31096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4cd0cfdea4a45fa82fbad0a9170c814: Sending DeleteTablet for 3 replicas of tablet 135567fe216e4a1d9356ed4d8afb9e23
I20260929 18:52:41.342691 31234 tablet_service.cc:1514] Processing DeleteTablet for tablet 135567fe216e4a1d9356ed4d8afb9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:41 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:36336
I20260929 18:52:41.342731 31513 tablet_service.cc:1514] Processing DeleteTablet for tablet 135567fe216e4a1d9356ed4d8afb9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:41 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:48058
I20260929 18:52:41.342804 31626 tablet_replica.cc:330] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: stopping tablet replica
I20260929 18:52:41.342823 31374 tablet_service.cc:1514] Processing DeleteTablet for tablet 135567fe216e4a1d9356ed4d8afb9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:41 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:49864
I20260929 18:52:41.342849 31626 raft_consensus.cc:2229] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:41.342880 31626 raft_consensus.cc:2258] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:41.342903 31628 tablet_replica.cc:330] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: stopping tablet replica
I20260929 18:52:41.342903 31627 tablet_replica.cc:330] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: stopping tablet replica
I20260929 18:52:41.342945 31628 raft_consensus.cc:2229] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:41.342952 31627 raft_consensus.cc:2229] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:41.342983 31627 raft_consensus.cc:2258] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:41.343027 31628 raft_consensus.cc:2258] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:41.343329 31628 ts_tablet_manager.cc:1905] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:41.343353 31627 ts_tablet_manager.cc:1905] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:41.343875 31626 ts_tablet_manager.cc:1905] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:41.345276 31627 ts_tablet_manager.cc:1918] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:41.345276 31628 ts_tablet_manager.cc:1918] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:41.345327 31627 log.cc:1196] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/135567fe216e4a1d9356ed4d8afb9e23
I20260929 18:52:41.345335 31628 log.cc:1196] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/135567fe216e4a1d9356ed4d8afb9e23
I20260929 18:52:41.345359 31626 ts_tablet_manager.cc:1918] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:41.345412 31626 log.cc:1196] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/135567fe216e4a1d9356ed4d8afb9e23
I20260929 18:52:41.345556 31628 ts_tablet_manager.cc:1939] T 135567fe216e4a1d9356ed4d8afb9e23 P 582b5de4e89e43b28bc5e2acb3139423: Deleting consensus metadata
I20260929 18:52:41.345556 31627 ts_tablet_manager.cc:1939] T 135567fe216e4a1d9356ed4d8afb9e23 P 63cd6b2b6cc64e03bb3974e2e8e8071c: Deleting consensus metadata
I20260929 18:52:41.345625 31626 ts_tablet_manager.cc:1939] T 135567fe216e4a1d9356ed4d8afb9e23 P ac261e49d28340f6b55836e9667562f3: Deleting consensus metadata
I20260929 18:52:41.345855 31084 catalog_manager.cc:4841] TS 582b5de4e89e43b28bc5e2acb3139423 (127.29.71.131:35265): tablet 135567fe216e4a1d9356ed4d8afb9e23 (table test-table [id=f0cc5035dba1447faf767b5fe61415a8]) successfully deleted
I20260929 18:52:41.345906 31082 catalog_manager.cc:4841] TS ac261e49d28340f6b55836e9667562f3 (127.29.71.129:44235): tablet 135567fe216e4a1d9356ed4d8afb9e23 (table test-table [id=f0cc5035dba1447faf767b5fe61415a8]) successfully deleted
I20260929 18:52:41.345947 31084 catalog_manager.cc:4841] TS 63cd6b2b6cc64e03bb3974e2e8e8071c (127.29.71.130:44515): tablet 135567fe216e4a1d9356ed4d8afb9e23 (table test-table [id=f0cc5035dba1447faf767b5fe61415a8]) successfully deleted
Sep 29 18:52:41 dist-test-slave-q2kh krb5kdc[31052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707960, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:52:41.355161 31600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.346390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36350 (local address 127.29.71.129:44235)
0929 18:52:41.346429 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.346431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.346567 (+ 136us) server_negotiation.cc:407] Connection header received
0929 18:52:41.346643 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.346644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.346656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.346711 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:41.347293 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.347777 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.348538 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.348753 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.348851 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.351957 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:41.351967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:41.351967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:41.351975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:41.353865 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:41.354293 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:41.354297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:41.354298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:41.354374 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:41.354601 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:41.354602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:41.354602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:41.354774 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:41.354916 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.354979 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.355064 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 18:52:41.355443 31294 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36350
I20260929 18:52:41.362543 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:41.356411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.29.71.190:36815)
0929 18:52:41.356459 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:41.356461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:41.356535 (+ 74us) server_negotiation.cc:407] Connection header received
0929 18:52:41.356611 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:41.356612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:41.356623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:41.356677 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:41.357183 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.357682 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.358520 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:41.358731 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:41.358821 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:41.360048 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:41.360049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:41.360050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:41.360056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:41.361383 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:41.361732 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:41.361733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:41.361733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:41.361793 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:41.361999 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:41.362000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:41.362000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:41.362116 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:41.362251 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:41.362357 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:52:41.362446 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 18:52:41.362812 31096 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40470
I20260929 18:52:41.363397 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 31169
I20260929 18:52:41.370191 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 31309
I20260929 18:52:41.377592 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 31449
I20260929 18:52:41.381980 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 31065
2026-09-29T18:52:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2655 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:41 dist-test-slave-q2kh krb5kdc[31665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:41 dist-test-slave-q2kh krb5kdc[31665](info): set up 2 sockets
Sep 29 18:52:41 dist-test-slave-q2kh krb5kdc[31665](info): commencing operation
krb5kdc: starting...
W20260929 18:52:42.434270 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:42 dist-test-slave-q2kh krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:42.562502 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34691
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:42.643994 31678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:42.644222 31678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:42.644245 31678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:42.645356 31678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:42.645391 31678 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:52:42.645404 31678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:42.645417 31678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:42.645429 31678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:42.645440 31678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:42.646972 31678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:34691
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:42.647171 31678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:42.647341 31678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:42.652465 31683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:42.652526 31684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:42.652508 31686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:42.652545 31678 server_base.cc:1034] running on GCE node
I20260929 18:52:42.652983 31678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:42.653287 31678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:42.654470 31678 hybrid_clock.cc:648] HybridClock initialized: now 1790707962654456 us; error 25 us; skew 500 ppm
Sep 29 18:52:42 dist-test-slave-q2kh krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:42.656857 31678 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:42.657438 31678 webserver.cc:460] Webserver started at http://127.29.71.190:44739/ using document root <none> and password file <none>
I20260929 18:52:42.657689 31678 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:42.657763 31678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:42.657927 31678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:42.659107 31678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "23c35e6b25ac40b7be591fc8c0a7b543"
format_stamp: "Formatted at 2026-09-29 18:52:42 on dist-test-slave-q2kh"
server_key: "85eec123d5fd99bb33b1d04c5219776b"
server_key_iv: "52014675ccf3d7b45de5bc43282e718d"
server_key_version: "encryptionkey@0"
I20260929 18:52:42.659528 31678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "23c35e6b25ac40b7be591fc8c0a7b543"
format_stamp: "Formatted at 2026-09-29 18:52:42 on dist-test-slave-q2kh"
server_key: "85eec123d5fd99bb33b1d04c5219776b"
server_key_iv: "52014675ccf3d7b45de5bc43282e718d"
server_key_version: "encryptionkey@0"
I20260929 18:52:42.660889 31678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:42.661628 31693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:42.661803 31678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:42.661911 31678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "23c35e6b25ac40b7be591fc8c0a7b543"
format_stamp: "Formatted at 2026-09-29 18:52:42 on dist-test-slave-q2kh"
server_key: "85eec123d5fd99bb33b1d04c5219776b"
server_key_iv: "52014675ccf3d7b45de5bc43282e718d"
server_key_version: "encryptionkey@0"
I20260929 18:52:42.662000 31678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:42.683467 31678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:42.685600 31678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:42.685801 31678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:42.690634 31678 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:34691
I20260929 18:52:42.690692 31755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:34691 every 8 connection(s)
I20260929 18:52:42.691020 31678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:42.691628 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:42.693985 31756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Bootstrap starting.
I20260929 18:52:42.694646 31756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:42.694949 31756 log.cc:824] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:42.695673 31756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: No bootstrap required, opened a new log
I20260929 18:52:42.697751 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 31678
I20260929 18:52:42.697841 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:42.698015 29982 external_mini_cluster.cc:1428] Setting key afc4eb09ffd7b391199bfa6678335d41
I20260929 18:52:42.697986 31756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34691 } }
I20260929 18:52:42.698251 31756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:42.698323 31756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c35e6b25ac40b7be591fc8c0a7b543, State: Initialized, Role: FOLLOWER
I20260929 18:52:42.698482 31756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34691 } }
I20260929 18:52:42.698688 31756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:42.698742 31756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:42.698791 31756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:42.699337 31756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34691 } }
I20260929 18:52:42.699462 31756 leader_election.cc:302] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c35e6b25ac40b7be591fc8c0a7b543; no voters:
I20260929 18:52:42.699672 31756 leader_election.cc:288] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:42.699690 31761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:42.699793 31761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [term 1 LEADER]: Becoming Leader. State: Replica: 23c35e6b25ac40b7be591fc8c0a7b543, State: Running, Role: LEADER
I20260929 18:52:42.699877 31761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34691 } }
I20260929 18:52:42.699937 31756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:42.700239 31761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34691 } } }
I20260929 18:52:42.700404 31761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:42.700590 31761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c35e6b25ac40b7be591fc8c0a7b543. Latest consensus state: current_term: 1 leader_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c35e6b25ac40b7be591fc8c0a7b543" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34691 } } }
I20260929 18:52:42.700692 31761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:42.700979 31769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:42.701592 31769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:42.703372 31769 catalog_manager.cc:1347] Generated new cluster ID: 54b11a9c399f4f52b8f6ab4186b3fdaa
I20260929 18:52:42.703428 31769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:52:42 dist-test-slave-q2kh krb5kdc[31665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:42.709686 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:42.698548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.29.71.190:34691)
0929 18:52:42.698746 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:42.698757 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:42.698791 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:52:42.698927 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:42.698940 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:42.698985 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:42.699134 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:42.699924 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.701414 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.702580 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.703109 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.703249 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:42.706102 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:42.706106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:42.706106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:42.706118 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:42.708039 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:42.708452 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:42.708455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:42.708455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:42.708573 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:42.708795 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:42.708795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:42.708796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:42.709081 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:42.709229 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:42.709358 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:52:42.709494 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260929 18:52:42.712147 31769 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:42.712644 31769 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:42.722997 31769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Generated new TSK 0
I20260929 18:52:42.723213 31769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:52:42.757114 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34691
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:42.840793 31782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:42.841060 31782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:42.841104 31782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:42.842422 31782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:42.842502 31782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:42.842537 31782 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:52:42.842566 31782 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:52:42.842595 31782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:42.844069 31782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:42.844365 31782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:42.844611 31782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:42.849995 31787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:42.850018 31790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:42.850023 31788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:42.850426 31782 server_base.cc:1034] running on GCE node
I20260929 18:52:42.850629 31782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:42.850908 31782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:42.852066 31782 hybrid_clock.cc:648] HybridClock initialized: now 1790707962852053 us; error 28 us; skew 500 ppm
Sep 29 18:52:42 dist-test-slave-q2kh krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:42.854651 31782 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:42.855165 31782 webserver.cc:460] Webserver started at http://127.29.71.129:34343/ using document root <none> and password file <none>
I20260929 18:52:42.855408 31782 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:42.855484 31782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:42.855623 31782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:42.856719 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "ffefa730ca9f4c4086bac0c64756aaf0"
format_stamp: "Formatted at 2026-09-29 18:52:42 on dist-test-slave-q2kh"
server_key: "7359c0da961f1a5408ab07c479674b5b"
server_key_iv: "eb401dcc2298738273a6f3a085cf35ab"
server_key_version: "encryptionkey@0"
I20260929 18:52:42.857108 31782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "ffefa730ca9f4c4086bac0c64756aaf0"
format_stamp: "Formatted at 2026-09-29 18:52:42 on dist-test-slave-q2kh"
server_key: "7359c0da961f1a5408ab07c479674b5b"
server_key_iv: "eb401dcc2298738273a6f3a085cf35ab"
server_key_version: "encryptionkey@0"
I20260929 18:52:42.858551 31782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:42.859306 31797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:42.859494 31782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:42.859596 31782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "ffefa730ca9f4c4086bac0c64756aaf0"
format_stamp: "Formatted at 2026-09-29 18:52:42 on dist-test-slave-q2kh"
server_key: "7359c0da961f1a5408ab07c479674b5b"
server_key_iv: "eb401dcc2298738273a6f3a085cf35ab"
server_key_version: "encryptionkey@0"
I20260929 18:52:42.859678 31782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:42.873399 31782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:42.875502 31782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:42.875672 31782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:42.876298 31782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:42.876349 31782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:42.876382 31782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:42.876400 31782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:52:42 dist-test-slave-q2kh krb5kdc[31665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:42.883819 31782 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:33565
I20260929 18:52:42.883937 31912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:33565 every 8 connection(s)
I20260929 18:52:42.884245 31782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:52:42.886449 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:42.876832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:41095 (local address 127.29.71.190:34691)
0929 18:52:42.876913 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:42.876915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:42.877225 (+ 310us) server_negotiation.cc:407] Connection header received
0929 18:52:42.878007 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:42.878009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:42.878023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:42.878094 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:42.878749 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.879271 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.880315 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.880520 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.880615 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:42.883447 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:42.883448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:42.883449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:42.883456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:42.885125 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:42.885508 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:42.885509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:42.885509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:42.885562 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:42.885795 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:42.885796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:42.885796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:42.885924 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:42.885990 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:42.886310 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 18:52:42.886382 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:52:42.886616 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:42.876846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34691 (local address 127.29.71.129:41095)
0929 18:52:42.877061 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:42.877114 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:42.877856 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:42.878201 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:42.878205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:42.878461 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:42.878644 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:42.878650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.879401 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.879402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:42.880212 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:42.880215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.880623 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.880624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:42.880804 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:42.881328 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:42.881349 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:42.883353 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:42.885201 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:42.885206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:42.885208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:42.885419 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:42.885619 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:42.885619 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:42.885620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:42.885708 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:42.886001 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:42.886007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:42.886119 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 18:52:42.886273 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 18:52:42.886406 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260929 18:52:42.887097 31913 heartbeater.cc:347] Connected to a master server at 127.29.71.190:34691
I20260929 18:52:42.887181 31913 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:42.887411 31913 heartbeater.cc:511] Master 127.29.71.190:34691 requested a full tablet report, sending...
I20260929 18:52:42.887990 31709 ts_manager.cc:194] Registered new tserver with Master: ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565)
I20260929 18:52:42.888809 31709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:41095
I20260929 18:52:42.892496 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 31782
I20260929 18:52:42.892627 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:42.892815 29982 external_mini_cluster.cc:1428] Setting key 5973eaf0bc35307e22812dee534d6171
I20260929 18:52:42.893648 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:42.888248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:50795 (local address 127.29.71.190:34691)
0929 18:52:42.888290 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:42.888292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:42.888359 (+ 67us) server_negotiation.cc:407] Connection header received
0929 18:52:42.888430 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:42.888431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:42.888441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:42.888495 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:42.888772 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.889427 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.890134 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.890388 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.890452 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:42.891324 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:42.891325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:42.891325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:42.891330 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:42.892472 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:42.892850 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:42.892851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:42.892851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:42.892928 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:42.893169 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:42.893170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:42.893171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:42.893289 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:42.893348 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:42.893482 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:52:42.893558 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:52:42.893671 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:42.888187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34691 (local address 127.29.71.129:50795)
0929 18:52:42.888300 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:42.888310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:42.888371 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:42.888510 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:42.888510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:42.888654 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:42.888702 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:42.888704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.889526 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.889527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:42.890048 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:42.890049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:42.890470 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:42.890471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:42.890529 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:42.890968 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:42.890972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:42.891247 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:42.892554 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:42.892556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:42.892556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:42.892774 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:42.893064 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:42.893065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:42.893065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:42.893119 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:42.893353 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:42.893354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:42.893391 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:52:42.893443 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:52:42.893571 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:42.922641 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34691
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:43.004106 31922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:43.004330 31922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:43.004356 31922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:43.005472 31922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:43.005508 31922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:43.005528 31922 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:52:43.005540 31922 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:52:43.005558 31922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:43.007140 31922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:43.007336 31922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:43.007522 31922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:43.012506 31927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:43.012655 31928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:43.012593 31930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:43.012780 31922 server_base.cc:1034] running on GCE node
I20260929 18:52:43.013016 31922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:43.013276 31922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:43.014447 31922 hybrid_clock.cc:648] HybridClock initialized: now 1790707963014419 us; error 38 us; skew 500 ppm
Sep 29 18:52:43 dist-test-slave-q2kh krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707963, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:43.017083 31922 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:43.017601 31922 webserver.cc:460] Webserver started at http://127.29.71.130:33277/ using document root <none> and password file <none>
I20260929 18:52:43.017844 31922 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:43.017920 31922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:43.018097 31922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:43.019212 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "4363ace3311843c09b62ff1363942478"
format_stamp: "Formatted at 2026-09-29 18:52:43 on dist-test-slave-q2kh"
server_key: "ce9308e3cefad9c62a086ef683540511"
server_key_iv: "8b1803d766ad76a4601c6a8eab6cf17f"
server_key_version: "encryptionkey@0"
I20260929 18:52:43.019601 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "4363ace3311843c09b62ff1363942478"
format_stamp: "Formatted at 2026-09-29 18:52:43 on dist-test-slave-q2kh"
server_key: "ce9308e3cefad9c62a086ef683540511"
server_key_iv: "8b1803d766ad76a4601c6a8eab6cf17f"
server_key_version: "encryptionkey@0"
I20260929 18:52:43.021029 31922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:43.021831 31937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:43.022043 31922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:43.022159 31922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "4363ace3311843c09b62ff1363942478"
format_stamp: "Formatted at 2026-09-29 18:52:43 on dist-test-slave-q2kh"
server_key: "ce9308e3cefad9c62a086ef683540511"
server_key_iv: "8b1803d766ad76a4601c6a8eab6cf17f"
server_key_version: "encryptionkey@0"
I20260929 18:52:43.022295 31922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:43.041421 31922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:43.043457 31922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:43.043644 31922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:43.044346 31922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:43.044399 31922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:43.044430 31922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:43.044452 31922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:52:43 dist-test-slave-q2kh krb5kdc[31665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707963, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:43.052067 31922 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:42411
I20260929 18:52:43.052153 32052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:42411 every 8 connection(s)
I20260929 18:52:43.052454 31922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:52:43.054893 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.044827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:42537 (local address 127.29.71.190:34691)
0929 18:52:43.044906 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.044909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.045187 (+ 278us) server_negotiation.cc:407] Connection header received
0929 18:52:43.045747 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.045749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.045763 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.045846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:43.046702 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.047389 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.048617 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.048956 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.049053 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.051997 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:43.051998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:43.051998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:43.052007 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:43.053631 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.053990 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.053990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.053991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.054041 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.054299 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.054300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.054300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.054404 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:43.054459 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.054710 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.054824 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:52:43.054975 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.044827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34691 (local address 127.29.71.130:42537)
0929 18:52:43.045031 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.045076 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.045635 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.045911 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.045915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.046381 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:43.046586 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.046592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.047507 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.047508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.048508 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.048510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.049090 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.049091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.049214 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.049826 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:43.049838 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:43.051886 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:43.053707 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.053711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.053712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.053918 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.054088 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.054088 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.054089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.054243 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.054468 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:43.054473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:43.054590 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.054667 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.054782 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260929 18:52:43.055461 32053 heartbeater.cc:347] Connected to a master server at 127.29.71.190:34691
I20260929 18:52:43.055538 32053 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:43.055718 32053 heartbeater.cc:511] Master 127.29.71.190:34691 requested a full tablet report, sending...
I20260929 18:52:43.056036 31709 ts_manager.cc:194] Registered new tserver with Master: 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411)
I20260929 18:52:43.056429 31709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:42537
I20260929 18:52:43.057976 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 31922
I20260929 18:52:43.058064 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:43.058279 29982 external_mini_cluster.cc:1428] Setting key e4b922c9e4d0f3ec002244dca97e2f3b
I20260929 18:52:43.062501 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.056602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:38649 (local address 127.29.71.190:34691)
0929 18:52:43.056646 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.056648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.056722 (+ 74us) server_negotiation.cc:407] Connection header received
0929 18:52:43.056794 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.056794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.056804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.056857 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:43.057175 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.057646 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.058567 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.058853 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.058913 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.059858 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:43.059859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:43.059860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:43.059867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:43.061260 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.061635 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.061637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.061637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.061694 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.061909 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.061910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.061911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.062069 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:43.062159 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.062331 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.062409 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:52:43.062544 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.056479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34691 (local address 127.29.71.130:38649)
0929 18:52:43.056672 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.056681 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.056737 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.056863 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.056864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.057036 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:43.057091 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.057093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.057748 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.057750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.058435 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.058456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.058935 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.058937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.059013 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.059477 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:43.059482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:43.059764 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:43.061340 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.061341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.061341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.061508 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.061779 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.061780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.061781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.061841 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.062148 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:43.062149 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:43.062219 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.062306 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.062411 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:43.086023 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34691
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:43.172114 32062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:43.172328 32062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:43.172355 32062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:43.173498 32062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:43.173544 32062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:43.173568 32062 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:52:43.173581 32062 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:52:43.173600 32062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:43.175118 32062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:43.175371 32062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:43.175613 32062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:43.180784 32070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:43.180931 32067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:43.180835 32068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:43.180869 32062 server_base.cc:1034] running on GCE node
I20260929 18:52:43.181208 32062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:43.181456 32062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:43.182629 32062 hybrid_clock.cc:648] HybridClock initialized: now 1790707963182626 us; error 57 us; skew 500 ppm
Sep 29 18:52:43 dist-test-slave-q2kh krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707963, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:43.185025 32062 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:43.185449 32062 webserver.cc:460] Webserver started at http://127.29.71.131:35615/ using document root <none> and password file <none>
I20260929 18:52:43.185678 32062 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:43.185750 32062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:43.185902 32062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:43.187024 32062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "a90fd353351744b38cbdc2f19bf67ef8"
format_stamp: "Formatted at 2026-09-29 18:52:43 on dist-test-slave-q2kh"
server_key: "0063def41b79adf55c38dc9ac4e412dc"
server_key_iv: "42932d9914f01f1ab5acf482b249e7e7"
server_key_version: "encryptionkey@0"
I20260929 18:52:43.187486 32062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "a90fd353351744b38cbdc2f19bf67ef8"
format_stamp: "Formatted at 2026-09-29 18:52:43 on dist-test-slave-q2kh"
server_key: "0063def41b79adf55c38dc9ac4e412dc"
server_key_iv: "42932d9914f01f1ab5acf482b249e7e7"
server_key_version: "encryptionkey@0"
I20260929 18:52:43.188861 32062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:52:43.189632 32077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:43.189920 32062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:43.190034 32062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "a90fd353351744b38cbdc2f19bf67ef8"
format_stamp: "Formatted at 2026-09-29 18:52:43 on dist-test-slave-q2kh"
server_key: "0063def41b79adf55c38dc9ac4e412dc"
server_key_iv: "42932d9914f01f1ab5acf482b249e7e7"
server_key_version: "encryptionkey@0"
I20260929 18:52:43.190111 32062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:43.199179 32062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:43.201671 32062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:43.201840 32062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:43.202491 32062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:43.202554 32062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:43.202620 32062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:43.202651 32062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:52:43 dist-test-slave-q2kh krb5kdc[31665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707963, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:43.210194 32062 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:36333
I20260929 18:52:43.210253 32192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:36333 every 8 connection(s)
I20260929 18:52:43.210664 32062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:52:43.211373 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 32062
I20260929 18:52:43.211449 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:43.211637 29982 external_mini_cluster.cc:1428] Setting key 2a49f4de315387df7612f6b0eece38f6
I20260929 18:52:43.213754 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.202956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:43093 (local address 127.29.71.190:34691)
0929 18:52:43.203033 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.203036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.203257 (+ 221us) server_negotiation.cc:407] Connection header received
0929 18:52:43.203744 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.203746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.203759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.203846 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:43.204761 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.205689 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.207319 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.207659 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.207753 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.210628 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:43.210630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:43.210630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:43.210640 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:43.212352 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.212751 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.212752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.212753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.212811 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.213085 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.213086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.213086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.213208 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:43.213293 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.213585 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.213665 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:52:43.213941 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.202943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34691 (local address 127.29.71.131:43093)
0929 18:52:43.203120 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.203159 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.203644 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.203945 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.203948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.204329 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:43.204645 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.204651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.205799 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.205800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.207207 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.207210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.207778 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.207779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.207929 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.208626 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:43.208635 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:43.210529 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:43.212442 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.212448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.212450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.212664 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.212893 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.212894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.212894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.213009 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.213281 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:43.213287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:43.213429 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.213558 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.213716 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":116,"threads_started":1}
I20260929 18:52:43.214481 32193 heartbeater.cc:347] Connected to a master server at 127.29.71.190:34691
I20260929 18:52:43.214572 32193 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:43.214751 32193 heartbeater.cc:511] Master 127.29.71.190:34691 requested a full tablet report, sending...
I20260929 18:52:43.215059 31709 ts_manager.cc:194] Registered new tserver with Master: a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:43.215533 31709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:43093
I20260929 18:52:43.220845 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.215624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34691 (local address 127.29.71.131:51419)
0929 18:52:43.215862 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.215871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.215919 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.216060 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.216061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.216210 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:43.216283 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.216285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.216931 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.216931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.217413 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.217414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.217718 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.217718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.217771 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.218123 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:43.218127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:43.218451 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:43.219971 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.219972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.219972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.220117 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.220390 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.220391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.220391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.220446 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.220631 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:43.220632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:43.220664 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.220713 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.220781 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":6,"thread_start_us":81,"threads_started":1}
I20260929 18:52:43.220870 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.215690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:51419 (local address 127.29.71.190:34691)
0929 18:52:43.215728 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.215730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.215937 (+ 207us) server_negotiation.cc:407] Connection header received
0929 18:52:43.215960 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.215961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.215971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.216049 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:43.216360 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.216849 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.217483 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.217653 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.217703 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.218540 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:43.218541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:43.218541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:43.218548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:43.219901 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.220242 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.220243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.220244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.220320 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.220485 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.220485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.220486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.220578 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:43.220631 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.220730 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.220813 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:52:43.223595 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:52:43 dist-test-slave-q2kh krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:52:43 dist-test-slave-q2kh krb5kdc[31665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:43.245142 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.238564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.29.71.190:34691)
0929 18:52:43.238630 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.238632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.238640 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:52:43.238661 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.238662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.238671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.238720 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:43.239258 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.239730 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.240424 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.240603 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.240651 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.242825 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:43.242827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:43.242827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:43.242834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:43.244270 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.244550 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.244550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.244551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.244594 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.244732 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.244732 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.244732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.244855 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:43.244928 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.245025 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.245079 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:52:43.246935 31709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:52:43.248517 31709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:43.255247 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.251186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.29.71.131:36333)
0929 18:52:43.251369 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.251374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.251384 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:43.251425 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.251428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.251445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.251551 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:43.251779 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.252920 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.254298 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.254847 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.254956 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.255019 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.255115 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.255152 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260929 18:52:43.255242 32211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.250939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:36333 (local address 127.0.0.1:42354)
0929 18:52:43.251102 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.251150 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.251249 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.251646 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.251648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.251651 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:43.251713 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.251715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.253033 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.253034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.254222 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.254225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.254960 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.254961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.255040 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.255044 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.255101 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.255145 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":61,"thread_start_us":62,"threads_started":1}
I20260929 18:52:43.255422 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.251252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.29.71.129:33565)
0929 18:52:43.251417 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.251421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.251429 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:52:43.251463 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.251466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.251481 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.251753 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:43.251808 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.252629 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.254032 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.254817 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.255197 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.255254 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.255277 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.255319 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260929 18:52:43.255455 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.250939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33565 (local address 127.0.0.1:59854)
0929 18:52:43.251126 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.251152 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.251218 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.251585 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.251588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.251590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:43.251675 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.251678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.252717 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.252718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.253937 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.253939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.254919 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.254920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.255012 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.255027 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.255080 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.255147 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260929 18:52:43.255649 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.251441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.29.71.130:42411)
0929 18:52:43.251848 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.251853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.251862 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:43.251898 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.251899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.251914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.251993 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:43.252114 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.253120 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.254382 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.255091 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.255152 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.255205 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.255510 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.255548 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":102,"threads_started":1}
I20260929 18:52:43.255755 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.250978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42411 (local address 127.0.0.1:53454)
0929 18:52:43.251623 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.251634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.251684 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.252010 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.252011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.252013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:43.252048 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.252049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.253210 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.253211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.254317 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.254319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.255239 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.255240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.255296 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.255299 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.255347 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.255386 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":111,"threads_started":1}
I20260929 18:52:43.256294 31847 tablet_service.cc:1467] Processing CreateTablet for tablet 22157d5d6a634853bf8f5c01841785f7 (DEFAULT_TABLE table=test-table [id=7c7ceee8a8734272929371c75b0c41e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:43.256457 31987 tablet_service.cc:1467] Processing CreateTablet for tablet 22157d5d6a634853bf8f5c01841785f7 (DEFAULT_TABLE table=test-table [id=7c7ceee8a8734272929371c75b0c41e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:43.256705 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22157d5d6a634853bf8f5c01841785f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.256760 31987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22157d5d6a634853bf8f5c01841785f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.260422 32216 tablet_bootstrap.cc:492] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Bootstrap starting.
I20260929 18:52:43.261047 32216 tablet_bootstrap.cc:654] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.261196 32127 tablet_service.cc:1467] Processing CreateTablet for tablet 22157d5d6a634853bf8f5c01841785f7 (DEFAULT_TABLE table=test-table [id=7c7ceee8a8734272929371c75b0c41e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:43.261382 32216 log.cc:824] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:43.261536 32127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22157d5d6a634853bf8f5c01841785f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.261758 32217 tablet_bootstrap.cc:492] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Bootstrap starting.
I20260929 18:52:43.262260 32216 tablet_bootstrap.cc:492] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: No bootstrap required, opened a new log
I20260929 18:52:43.262331 32216 ts_tablet_manager.cc:1397] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:52:43.262567 32217 tablet_bootstrap.cc:654] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.262888 32217 log.cc:824] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:43.263691 32217 tablet_bootstrap.cc:492] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: No bootstrap required, opened a new log
I20260929 18:52:43.263764 32217 ts_tablet_manager.cc:1397] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:52:43.264544 32216 raft_consensus.cc:348] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.264679 32220 tablet_bootstrap.cc:492] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Bootstrap starting.
I20260929 18:52:43.264716 32216 raft_consensus.cc:374] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.264816 32216 raft_consensus.cc:729] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffefa730ca9f4c4086bac0c64756aaf0, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.264937 32216 consensus_queue.cc:260] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.265219 32216 ts_tablet_manager.cc:1428] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:43.265340 31913 heartbeater.cc:503] Master 127.29.71.190:34691 was elected leader, sending a full tablet report...
I20260929 18:52:43.265739 32220 tablet_bootstrap.cc:654] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.265848 32217 raft_consensus.cc:348] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.266026 32217 raft_consensus.cc:374] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.266081 32217 raft_consensus.cc:729] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4363ace3311843c09b62ff1363942478, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.266081 32220 log.cc:824] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:43.266235 32217 consensus_queue.cc:260] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.266527 32217 ts_tablet_manager.cc:1428] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:43.266552 32053 heartbeater.cc:503] Master 127.29.71.190:34691 was elected leader, sending a full tablet report...
I20260929 18:52:43.266954 32220 tablet_bootstrap.cc:492] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: No bootstrap required, opened a new log
I20260929 18:52:43.267052 32220 ts_tablet_manager.cc:1397] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:52:43.268203 32221 raft_consensus.cc:482] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:43.268301 32221 raft_consensus.cc:504] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.268601 32221 leader_election.cc:288] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:43.268543 32220 raft_consensus.cc:348] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.268728 32220 raft_consensus.cc:374] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.268767 32220 raft_consensus.cc:729] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90fd353351744b38cbdc2f19bf67ef8, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.269088 32220 consensus_queue.cc:260] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.269374 32193 heartbeater.cc:503] Master 127.29.71.190:34691 was elected leader, sending a full tablet report...
I20260929 18:52:43.269356 32220 ts_tablet_manager.cc:1428] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:52:43.271793 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.268754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42411 (local address 127.29.71.129:50905)
0929 18:52:43.268797 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.268810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.268853 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.269028 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.269029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.269030 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:43.269094 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.269095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.269892 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.269893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.270947 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.270949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.271531 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.271531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.271617 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.271620 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.271670 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.271707 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 18:52:43.271796 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.268867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:50905 (local address 127.29.71.130:42411)
0929 18:52:43.268913 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.268915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.268925 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:43.268949 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.268949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.268959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.269018 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:43.269147 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.269801 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.271060 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.271423 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.271531 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.271540 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.271687 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.271728 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:52:43.271955 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.268822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:45391 (local address 127.29.71.131:36333)
0929 18:52:43.268924 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.268926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.269034 (+ 108us) server_negotiation.cc:407] Connection header received
0929 18:52:43.269140 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.269141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.269150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.269178 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:43.269409 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.269974 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.271110 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.271476 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.271610 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.271620 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.271834 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.271878 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 18:52:43.272174 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.268756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:36333 (local address 127.29.71.129:45391)
0929 18:52:43.268969 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.268979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.269068 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.269260 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.269260 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.269262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:43.269331 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.269333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.270102 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.270103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.271026 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.271028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.271607 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.271608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.271748 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.271751 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.272062 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.272099 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":86,"thread_start_us":83,"threads_started":1}
I20260929 18:52:43.272195 32007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22157d5d6a634853bf8f5c01841785f7" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4363ace3311843c09b62ff1363942478" is_pre_election: true
I20260929 18:52:43.272328 32007 raft_consensus.cc:2454] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 0.
I20260929 18:52:43.272550 31798 leader_election.cc:302] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, ffefa730ca9f4c4086bac0c64756aaf0; no voters:
I20260929 18:52:43.272544 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22157d5d6a634853bf8f5c01841785f7" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8" is_pre_election: true
I20260929 18:52:43.272655 32147 raft_consensus.cc:2454] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 0.
I20260929 18:52:43.272688 32221 raft_consensus.cc:2781] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:43.272756 32221 raft_consensus.cc:482] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:43.272786 32221 raft_consensus.cc:3037] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.273423 32221 raft_consensus.cc:504] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.273589 32221 leader_election.cc:288] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:43.273743 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22157d5d6a634853bf8f5c01841785f7" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8"
I20260929 18:52:43.273777 32007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22157d5d6a634853bf8f5c01841785f7" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4363ace3311843c09b62ff1363942478"
I20260929 18:52:43.273808 32147 raft_consensus.cc:3037] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.273880 32007 raft_consensus.cc:3037] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.274526 32147 raft_consensus.cc:2454] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 1.
I20260929 18:52:43.274724 32007 raft_consensus.cc:2454] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 1.
I20260929 18:52:43.275054 31799 leader_election.cc:302] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a90fd353351744b38cbdc2f19bf67ef8, ffefa730ca9f4c4086bac0c64756aaf0; no voters:
I20260929 18:52:43.275187 32221 raft_consensus.cc:2781] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:43.275319 32221 raft_consensus.cc:686] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 LEADER]: Becoming Leader. State: Replica: ffefa730ca9f4c4086bac0c64756aaf0, State: Running, Role: LEADER
I20260929 18:52:43.275398 32221 consensus_queue.cc:237] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.276225 31710 catalog_manager.cc:5495] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129). New cstate: current_term: 1 leader_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:52:43 dist-test-slave-q2kh krb5kdc[31665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707962, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:43.301779 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.294945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.29.71.190:34691)
0929 18:52:43.295041 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.295043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.295092 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:52:43.295151 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.295152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.295161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.295202 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:43.295653 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.296160 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.296854 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.297036 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.297087 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.299022 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:43.299024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:43.299024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:43.299030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:43.300553 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.301038 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.301038 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.301039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.301089 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:43.301299 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:43.301300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:43.301300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:43.301399 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:43.301482 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.301644 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.301714 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:52:43.301928 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.294859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34691 (local address 127.0.0.1:44994)
0929 18:52:43.295014 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.295024 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.295094 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.295228 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.295229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.295530 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:43.295576 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.295578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.296235 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.296236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.296776 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.296778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.297102 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.297102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.297141 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.297620 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:43.297624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:43.298940 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:43.300636 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.300643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.300645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.300952 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.301175 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:43.301176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:43.301176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:43.301237 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:43.301471 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:43.301477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:43.301569 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.301626 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.301785 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":31,"thread_start_us":76,"threads_started":1}
W20260929 18:52:43.303105 32054 tablet.cc:2369] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:52:43.303951 31710 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:44994:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:52:43.304195 31710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:43.305763 31987 tablet_service.cc:1467] Processing CreateTablet for tablet 502c6269faad4cb8acc3601e98af661c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=994b40617b3449f788391a4bdebb317f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:52:43.305794 32127 tablet_service.cc:1467] Processing CreateTablet for tablet 502c6269faad4cb8acc3601e98af661c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=994b40617b3449f788391a4bdebb317f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:52:43.305935 31987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502c6269faad4cb8acc3601e98af661c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.305945 32127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502c6269faad4cb8acc3601e98af661c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.307286 32217 tablet_bootstrap.cc:492] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: Bootstrap starting.
I20260929 18:52:43.307389 32220 tablet_bootstrap.cc:492] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: Bootstrap starting.
I20260929 18:52:43.307402 31847 tablet_service.cc:1467] Processing CreateTablet for tablet 502c6269faad4cb8acc3601e98af661c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=994b40617b3449f788391a4bdebb317f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:52:43.307572 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502c6269faad4cb8acc3601e98af661c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.307874 32220 tablet_bootstrap.cc:654] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.307992 32217 tablet_bootstrap.cc:654] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.308502 32220 tablet_bootstrap.cc:492] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: No bootstrap required, opened a new log
I20260929 18:52:43.308558 32220 ts_tablet_manager.cc:1397] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:43.308660 32217 tablet_bootstrap.cc:492] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: No bootstrap required, opened a new log
I20260929 18:52:43.308734 32217 ts_tablet_manager.cc:1397] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:43.308703 32220 raft_consensus.cc:348] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.308774 32220 raft_consensus.cc:374] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.308799 32220 raft_consensus.cc:729] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90fd353351744b38cbdc2f19bf67ef8, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.308848 32220 consensus_queue.cc:260] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.308897 32217 raft_consensus.cc:348] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.308943 32220 ts_tablet_manager.cc:1428] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:43.308979 32217 raft_consensus.cc:374] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.309016 32217 raft_consensus.cc:729] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4363ace3311843c09b62ff1363942478, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.309063 32217 consensus_queue.cc:260] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.309098 32225 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.309163 32217 ts_tablet_manager.cc:1428] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:43.309171 32225 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:43.309269 32216 tablet_bootstrap.cc:492] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: Bootstrap starting.
I20260929 18:52:43.309294 32223 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.309381 32223 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:43.309726 32216 tablet_bootstrap.cc:654] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.310439 32216 tablet_bootstrap.cc:492] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: No bootstrap required, opened a new log
I20260929 18:52:43.310480 32216 ts_tablet_manager.cc:1397] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:43.310645 32216 raft_consensus.cc:348] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.310707 32216 raft_consensus.cc:374] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.310752 32216 raft_consensus.cc:729] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffefa730ca9f4c4086bac0c64756aaf0, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.310798 32216 consensus_queue.cc:260] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.310889 32216 ts_tablet_manager.cc:1428] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:43.310969 32221 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.311079 32221 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:43.355348 32223 raft_consensus.cc:482] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:43.355459 32223 raft_consensus.cc:504] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.355957 32223 leader_election.cc:288] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:43.358935 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.356271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:51537 (local address 127.29.71.129:33565)
0929 18:52:43.356312 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.356314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.356322 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:52:43.356352 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.356353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.356363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.356421 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:43.356643 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.357115 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.358107 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.358543 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.358632 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.358641 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.358830 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.358870 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:52:43.358940 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.356166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33565 (local address 127.29.71.130:51537)
0929 18:52:43.356209 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.356222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.356272 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.356479 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.356480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.356482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:43.356578 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.356581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.357223 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.357224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.357983 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.357985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.358642 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.358643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.358733 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.358735 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.358822 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.358865 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":20}
I20260929 18:52:43.358932 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.356243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:38671 (local address 127.29.71.131:36333)
0929 18:52:43.356288 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.356290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.356434 (+ 144us) server_negotiation.cc:407] Connection header received
0929 18:52:43.356470 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.356470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.356479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.356568 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:43.356785 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.357246 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.358182 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.358535 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.358626 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.358636 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.358821 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.358860 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:52:43.358955 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.356176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:36333 (local address 127.29.71.130:38671)
0929 18:52:43.356386 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.356393 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.356431 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.356626 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.356627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.356629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:43.356703 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.356704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.357336 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.357338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.358053 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.358055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.358656 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.358657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.358727 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.358730 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.358814 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.358857 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":16,"thread_start_us":120,"threads_started":1}
I20260929 18:52:43.359330 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c6269faad4cb8acc3601e98af661c" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8" is_pre_election: true
I20260929 18:52:43.359304 31867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c6269faad4cb8acc3601e98af661c" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" is_pre_election: true
I20260929 18:52:43.359439 32147 raft_consensus.cc:2454] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 0.
I20260929 18:52:43.359452 31867 raft_consensus.cc:2454] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 0.
I20260929 18:52:43.359671 31941 leader_election.cc:302] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, ffefa730ca9f4c4086bac0c64756aaf0; no voters:
I20260929 18:52:43.359831 32223 raft_consensus.cc:2781] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:43.359874 32223 raft_consensus.cc:482] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:43.359896 32223 raft_consensus.cc:3037] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.360559 32223 raft_consensus.cc:504] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.360719 32223 leader_election.cc:288] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 election: Requested vote from peers ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:43.360917 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c6269faad4cb8acc3601e98af661c" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8"
I20260929 18:52:43.360904 31867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502c6269faad4cb8acc3601e98af661c" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffefa730ca9f4c4086bac0c64756aaf0"
I20260929 18:52:43.360991 31867 raft_consensus.cc:3037] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.360997 32147 raft_consensus.cc:3037] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.361824 31867 raft_consensus.cc:2454] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 1.
I20260929 18:52:43.361824 32147 raft_consensus.cc:2454] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 1.
I20260929 18:52:43.362052 31939 leader_election.cc:302] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, a90fd353351744b38cbdc2f19bf67ef8; no voters:
I20260929 18:52:43.362186 32223 raft_consensus.cc:2781] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:43.362408 32223 raft_consensus.cc:686] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [term 1 LEADER]: Becoming Leader. State: Replica: 4363ace3311843c09b62ff1363942478, State: Running, Role: LEADER
I20260929 18:52:43.362545 32223 consensus_queue.cc:237] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.363072 32238 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: TxnStatusTablet state changed. Reason: New leader 4363ace3311843c09b62ff1363942478. Latest consensus state: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.363160 32238 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:52:43.363397 31710 catalog_manager.cc:5495] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4363ace3311843c09b62ff1363942478 (127.29.71.130). New cstate: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 18:52:43.363751 32240 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:52:43.363874 32240 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:52:43.384353 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.381490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.29.71.130:42411)
0929 18:52:43.381570 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.381571 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.381580 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:43.381648 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.381649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.381664 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.381720 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:43.381848 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.382462 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.383570 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.383747 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.383797 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.384022 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.384125 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.384187 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.384259 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.384278 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:52:43.384392 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.381423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42411 (local address 127.0.0.1:53462)
0929 18:52:43.381509 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.381521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.381565 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.381733 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.381734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.381736 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:52:43.381785 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.381787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.382547 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.382549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.383482 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.383485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.383825 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.383826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.383896 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.383915 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.384197 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.384205 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.384258 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.384308 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 18:52:43.385663 31914 tablet.cc:2369] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:52:43.386401 32147 raft_consensus.cc:1261] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Refusing update from remote peer 4363ace3311843c09b62ff1363942478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:43.386518 31867 raft_consensus.cc:1261] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Refusing update from remote peer 4363ace3311843c09b62ff1363942478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:43.386732 32238 consensus_queue.cc:1035] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:43.386837 32223 consensus_queue.cc:1035] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:43.387908 32221 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: TxnStatusTablet state changed. Reason: New leader 4363ace3311843c09b62ff1363942478. Latest consensus state: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.387995 32221 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:43.388099 32225 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: TxnStatusTablet state changed. Reason: New leader 4363ace3311843c09b62ff1363942478. Latest consensus state: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.388197 32225 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:43.389117 32238 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.389202 32238 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:52:43.389190 32221 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.389278 32221 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P ffefa730ca9f4c4086bac0c64756aaf0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:43.389338 32244 mvcc.cc:205] Tried to move back new op lower bound from 7334739818028138496 to 7334739817933623296. Current Snapshot: MvccSnapshot[applied={T|T < 7334739818028138496}]
I20260929 18:52:43.389112 32223 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.389498 32223 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P 4363ace3311843c09b62ff1363942478: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:52:43.389813 32225 tablet_replica.cc:439] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } } }
I20260929 18:52:43.389881 32225 tablet_replica.cc:442] T 502c6269faad4cb8acc3601e98af661c P a90fd353351744b38cbdc2f19bf67ef8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:52:43.389994 32245 mvcc.cc:205] Tried to move back new op lower bound from 7334739818028138496 to 7334739817933623296. Current Snapshot: MvccSnapshot[applied={T|T < 7334739818028138496}]
I20260929 18:52:43.394559 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.391731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.29.71.129:33565)
0929 18:52:43.391767 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.391769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.391778 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:43.391839 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.391840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.391854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.391906 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:43.392063 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.392615 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.393760 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.394021 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.394079 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.394267 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.394357 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.394416 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.394487 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.394508 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:52:43.399564 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.397053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:59747 (local address 127.29.71.130:42411)
0929 18:52:43.397095 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.397097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.397238 (+ 141us) server_negotiation.cc:407] Connection header received
0929 18:52:43.397304 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.397305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.397315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.397370 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:43.397511 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.397959 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.398805 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.399013 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.399104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.399298 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.399332 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.399402 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.399463 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.399482 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:52:43.399577 32255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.396980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42411 (local address 127.29.71.129:59747)
0929 18:52:43.397165 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.397174 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.397252 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.397373 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.397374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.397375 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:52:43.397428 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.397429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.398049 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.398051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.398698 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.398700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.399104 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.399105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.399171 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.399189 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.399402 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.399404 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.399445 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.399492 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260929 18:52:43.402092 32147 raft_consensus.cc:1261] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Refusing update from remote peer ffefa730ca9f4c4086bac0c64756aaf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:43.402225 32007 raft_consensus.cc:1261] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Refusing update from remote peer ffefa730ca9f4c4086bac0c64756aaf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:43.402500 32226 consensus_queue.cc:1035] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:43.402649 32221 consensus_queue.cc:1035] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:43.404366 32243 mvcc.cc:205] Tried to move back new op lower bound from 7334739818093047808 to 7334739817576734720. Current Snapshot: MvccSnapshot[applied={T|T < 7334739818093047808}]
I20260929 18:52:43.411653 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.408988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:48413 (local address 127.29.71.129:33565)
0929 18:52:43.409028 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.409030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.409039 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:43.409068 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.409069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.409079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.409134 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:43.409336 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.409815 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.410955 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.411140 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.411227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.411409 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.411444 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.411489 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.411559 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.411580 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:52:43.411661 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.408754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33565 (local address 127.29.71.130:48413)
0929 18:52:43.408867 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.408879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.408919 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.409181 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.409183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.409184 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:52:43.409252 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.409253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.409905 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.409906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.410867 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.410869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.411228 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.411229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.411293 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.411309 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.411509 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.411511 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.411557 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.411598 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 18:52:43.420851 31710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:43.420977 31710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:43.421774 31710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Sending DeleteTablet for 3 replicas of tablet 22157d5d6a634853bf8f5c01841785f7
I20260929 18:52:43.422079 31847 tablet_service.cc:1514] Processing DeleteTablet for tablet 22157d5d6a634853bf8f5c01841785f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:43 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59854
I20260929 18:52:43.422142 32127 tablet_service.cc:1514] Processing DeleteTablet for tablet 22157d5d6a634853bf8f5c01841785f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:43 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:42354
I20260929 18:52:43.422509 31987 tablet_service.cc:1514] Processing DeleteTablet for tablet 22157d5d6a634853bf8f5c01841785f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:43 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:53454
I20260929 18:52:43.422719 32263 tablet_replica.cc:330] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: stopping tablet replica
I20260929 18:52:43.422714 31710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:52:43.422816 32263 raft_consensus.cc:2229] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 18:52:43.422853 31710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:43.422864 32263 raft_consensus.cc:2258] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:43.423285 32263 ts_tablet_manager.cc:1905] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:43.424013 32127 tablet_service.cc:1467] Processing CreateTablet for tablet cda4f9f8db614e57937362295acd940d (DEFAULT_TABLE table=test-table [id=edc48ffea5af4d71b5fb3925dc96e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:43.424052 31847 tablet_service.cc:1467] Processing CreateTablet for tablet cda4f9f8db614e57937362295acd940d (DEFAULT_TABLE table=test-table [id=edc48ffea5af4d71b5fb3925dc96e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:43.424129 32127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda4f9f8db614e57937362295acd940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.424170 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda4f9f8db614e57937362295acd940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.425438 32220 tablet_bootstrap.cc:492] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Bootstrap starting.
I20260929 18:52:43.425535 32216 tablet_bootstrap.cc:492] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: Bootstrap starting.
I20260929 18:52:43.425806 32220 tablet_bootstrap.cc:654] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.425834 31987 tablet_service.cc:1467] Processing CreateTablet for tablet cda4f9f8db614e57937362295acd940d (DEFAULT_TABLE table=test-table [id=edc48ffea5af4d71b5fb3925dc96e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:43.425927 31987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda4f9f8db614e57937362295acd940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:43.425987 32216 tablet_bootstrap.cc:654] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.427271 32217 tablet_bootstrap.cc:492] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: Bootstrap starting.
I20260929 18:52:43.427812 32217 tablet_bootstrap.cc:654] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:43.428439 32217 tablet_bootstrap.cc:492] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: No bootstrap required, opened a new log
I20260929 18:52:43.428493 32217 ts_tablet_manager.cc:1397] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:43.428622 32217 raft_consensus.cc:348] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.428694 32217 raft_consensus.cc:374] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.428714 32217 raft_consensus.cc:729] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4363ace3311843c09b62ff1363942478, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.428757 32217 consensus_queue.cc:260] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.428844 32217 ts_tablet_manager.cc:1428] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:43.429145 32263 ts_tablet_manager.cc:1918] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:43.429219 32263 log.cc:1196] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/22157d5d6a634853bf8f5c01841785f7
I20260929 18:52:43.429492 32263 ts_tablet_manager.cc:1939] T 22157d5d6a634853bf8f5c01841785f7 P 4363ace3311843c09b62ff1363942478: Deleting consensus metadata
I20260929 18:52:43.429872 31694 catalog_manager.cc:4841] TS 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411): tablet 22157d5d6a634853bf8f5c01841785f7 (table test-table [id=7c7ceee8a8734272929371c75b0c41e9]) successfully deleted
I20260929 18:52:43.430253 32262 tablet_replica.cc:330] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: stopping tablet replica
I20260929 18:52:43.430325 32220 tablet_bootstrap.cc:492] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: No bootstrap required, opened a new log
I20260929 18:52:43.430373 32220 ts_tablet_manager.cc:1397] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 18:52:43.430459 32261 tablet_replica.cc:330] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: stopping tablet replica
I20260929 18:52:43.430557 32261 raft_consensus.cc:2229] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:43.430531 32220 raft_consensus.cc:348] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.430598 32220 raft_consensus.cc:374] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.430621 32220 raft_consensus.cc:729] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90fd353351744b38cbdc2f19bf67ef8, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.430657 32261 raft_consensus.cc:2258] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:43.430662 32220 consensus_queue.cc:260] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.430743 32220 ts_tablet_manager.cc:1428] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:43.430372 32262 raft_consensus.cc:2229] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:43.431088 32261 ts_tablet_manager.cc:1905] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:43.431195 32262 raft_consensus.cc:2258] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:43.431668 32262 ts_tablet_manager.cc:1905] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:43.432866 32261 ts_tablet_manager.cc:1918] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:43.432945 32261 log.cc:1196] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/22157d5d6a634853bf8f5c01841785f7
I20260929 18:52:43.433213 32261 ts_tablet_manager.cc:1939] T 22157d5d6a634853bf8f5c01841785f7 P ffefa730ca9f4c4086bac0c64756aaf0: Deleting consensus metadata
I20260929 18:52:43.433625 32262 ts_tablet_manager.cc:1918] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:43.433740 32262 log.cc:1196] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/22157d5d6a634853bf8f5c01841785f7
I20260929 18:52:43.433835 31697 catalog_manager.cc:4841] TS ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565): tablet 22157d5d6a634853bf8f5c01841785f7 (table test-table [id=7c7ceee8a8734272929371c75b0c41e9]) successfully deleted
I20260929 18:52:43.434096 32262 ts_tablet_manager.cc:1939] T 22157d5d6a634853bf8f5c01841785f7 P a90fd353351744b38cbdc2f19bf67ef8: Deleting consensus metadata
I20260929 18:52:43.434563 31695 catalog_manager.cc:4841] TS a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333): tablet 22157d5d6a634853bf8f5c01841785f7 (table test-table [id=7c7ceee8a8734272929371c75b0c41e9]) successfully deleted
I20260929 18:52:43.438678 32216 tablet_bootstrap.cc:492] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: No bootstrap required, opened a new log
I20260929 18:52:43.438742 32216 ts_tablet_manager.cc:1397] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20260929 18:52:43.438891 32216 raft_consensus.cc:348] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.438964 32216 raft_consensus.cc:374] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:43.438982 32216 raft_consensus.cc:729] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffefa730ca9f4c4086bac0c64756aaf0, State: Initialized, Role: FOLLOWER
I20260929 18:52:43.439039 32216 consensus_queue.cc:260] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.439126 32216 ts_tablet_manager.cc:1428] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260929 18:52:43.461416 32194 tablet.cc:2369] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:52:43.568393 32249 raft_consensus.cc:482] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:43.568495 32249 raft_consensus.cc:504] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.568691 32249 leader_election.cc:288] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:43.569072 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda4f9f8db614e57937362295acd940d" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8" is_pre_election: true
I20260929 18:52:43.569090 31867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda4f9f8db614e57937362295acd940d" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" is_pre_election: true
I20260929 18:52:43.569175 32147 raft_consensus.cc:2454] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 0.
I20260929 18:52:43.569218 31867 raft_consensus.cc:2454] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 0.
I20260929 18:52:43.569360 31939 leader_election.cc:302] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, a90fd353351744b38cbdc2f19bf67ef8; no voters:
I20260929 18:52:43.569479 32249 raft_consensus.cc:2781] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:43.569536 32249 raft_consensus.cc:482] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:43.569572 32249 raft_consensus.cc:3037] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.570351 32249 raft_consensus.cc:504] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.570519 32249 leader_election.cc:288] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 election: Requested vote from peers ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:43.570685 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda4f9f8db614e57937362295acd940d" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8"
I20260929 18:52:43.570710 31867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda4f9f8db614e57937362295acd940d" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffefa730ca9f4c4086bac0c64756aaf0"
I20260929 18:52:43.570788 31867 raft_consensus.cc:3037] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.570791 32147 raft_consensus.cc:3037] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:43.571591 31867 raft_consensus.cc:2454] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 1.
I20260929 18:52:43.571612 32147 raft_consensus.cc:2454] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 1.
I20260929 18:52:43.571756 31941 leader_election.cc:302] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, ffefa730ca9f4c4086bac0c64756aaf0; no voters:
I20260929 18:52:43.571862 32249 raft_consensus.cc:2781] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:43.571928 32249 raft_consensus.cc:686] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 1 LEADER]: Becoming Leader. State: Replica: 4363ace3311843c09b62ff1363942478, State: Running, Role: LEADER
I20260929 18:52:43.571982 32249 consensus_queue.cc:237] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:43.572584 31709 catalog_manager.cc:5495] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4363ace3311843c09b62ff1363942478 (127.29.71.130). New cstate: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 18:52:43.613062 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.611096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.29.71.130:42411)
0929 18:52:43.611135 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.611136 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.611149 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:52:43.611196 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.611197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.611204 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.611254 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:43.611335 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.611786 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.612516 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.612683 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.612732 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.612866 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.612892 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.612929 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.612979 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.612993 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 18:52:43.617398 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.615129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42411 (local address 127.29.71.130:45919)
0929 18:52:43.615186 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:43.615198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:43.615246 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:43.615391 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:43.615392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:43.615393 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:52:43.615426 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.615428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.616067 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.616067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.616644 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:43.616645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.617006 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.617007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:43.617054 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.617058 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.617240 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.617241 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 18:52:43.617294 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:52:43.617334 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 18:52:43.617398 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:43.615189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:45919 (local address 127.29.71.130:42411)
0929 18:52:43.615217 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:43.615218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:43.615230 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:52:43.615304 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:43.615304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:43.615313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:43.615387 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:43.615498 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.615927 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.616746 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:43.616920 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:43.617015 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:43.617143 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:43.617177 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:43.617241 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:43.617298 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:52:43.617314 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 18:52:43.619217 31867 raft_consensus.cc:1261] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Refusing update from remote peer 4363ace3311843c09b62ff1363942478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:43.619446 32147 raft_consensus.cc:1261] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Refusing update from remote peer 4363ace3311843c09b62ff1363942478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:43.619464 32249 consensus_queue.cc:1035] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:43.619669 32249 consensus_queue.cc:1035] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:52:43.624024 32243 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:52:45.143011 31709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:45.143141 31709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:45.143968 31709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Sending DeleteTablet for 3 replicas of tablet cda4f9f8db614e57937362295acd940d
I20260929 18:52:45.144236 31847 tablet_service.cc:1514] Processing DeleteTablet for tablet cda4f9f8db614e57937362295acd940d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:45 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59854
I20260929 18:52:45.144289 32127 tablet_service.cc:1514] Processing DeleteTablet for tablet cda4f9f8db614e57937362295acd940d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:45 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:42354
I20260929 18:52:45.144441 31986 tablet_service.cc:1514] Processing DeleteTablet for tablet cda4f9f8db614e57937362295acd940d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:45 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:53454
I20260929 18:52:45.144630 32294 tablet_replica.cc:330] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: stopping tablet replica
I20260929 18:52:45.144701 32294 raft_consensus.cc:2229] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:45.144824 32294 raft_consensus.cc:2258] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:45.145148 32294 ts_tablet_manager.cc:1905] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:45.147190 32294 ts_tablet_manager.cc:1918] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:52:45.147274 32294 log.cc:1196] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/cda4f9f8db614e57937362295acd940d
I20260929 18:52:45.147349 32292 tablet_replica.cc:330] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: stopping tablet replica
I20260929 18:52:45.147425 32292 raft_consensus.cc:2229] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:45.147475 32292 raft_consensus.cc:2258] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:45.147586 32293 tablet_replica.cc:330] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: stopping tablet replica
I20260929 18:52:45.147614 32294 ts_tablet_manager.cc:1939] T cda4f9f8db614e57937362295acd940d P 4363ace3311843c09b62ff1363942478: Deleting consensus metadata
I20260929 18:52:45.147711 32293 raft_consensus.cc:2229] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:45.147758 32292 ts_tablet_manager.cc:1905] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:45.147805 32293 raft_consensus.cc:2258] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:45.148015 31694 catalog_manager.cc:4841] TS 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411): tablet cda4f9f8db614e57937362295acd940d (table test-table [id=edc48ffea5af4d71b5fb3925dc96e03c]) successfully deleted
I20260929 18:52:45.148099 32293 ts_tablet_manager.cc:1905] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:45.149348 32292 ts_tablet_manager.cc:1918] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:52:45.149426 32292 log.cc:1196] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/cda4f9f8db614e57937362295acd940d
I20260929 18:52:45.149735 32292 ts_tablet_manager.cc:1939] T cda4f9f8db614e57937362295acd940d P ffefa730ca9f4c4086bac0c64756aaf0: Deleting consensus metadata
I20260929 18:52:45.149785 32293 ts_tablet_manager.cc:1918] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:52:45.149893 32293 log.cc:1196] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/cda4f9f8db614e57937362295acd940d
I20260929 18:52:45.150082 31697 catalog_manager.cc:4841] TS ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565): tablet cda4f9f8db614e57937362295acd940d (table test-table [id=edc48ffea5af4d71b5fb3925dc96e03c]) successfully deleted
I20260929 18:52:45.150295 32293 ts_tablet_manager.cc:1939] T cda4f9f8db614e57937362295acd940d P a90fd353351744b38cbdc2f19bf67ef8: Deleting consensus metadata
I20260929 18:52:45.150645 31695 catalog_manager.cc:4841] TS a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333): tablet cda4f9f8db614e57937362295acd940d (table test-table [id=edc48ffea5af4d71b5fb3925dc96e03c]) successfully deleted
Sep 29 18:52:45 dist-test-slave-q2kh krb5kdc[31665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:52:45 dist-test-slave-q2kh krb5kdc[31665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707965, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:45.176123 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:45.163441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.29.71.190:34691)
0929 18:52:45.163620 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:45.163622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:45.163632 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:45.163656 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:45.163657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:45.163667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:45.163728 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:45.164281 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:45.164806 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:45.165704 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:45.165922 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:45.166039 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:45.173424 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:45.173426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:45.173426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:45.173434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:45.174907 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:45.175318 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:45.175319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:45.175319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:45.175377 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:45.175592 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:45.175592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:45.175593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:45.175711 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:45.175824 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:45.175915 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:52:45.176008 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260929 18:52:45.177906 31709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:52:45.178082 31709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:45.179558 31847 tablet_service.cc:1467] Processing CreateTablet for tablet 9e950235fde54623bb477d0df660252f (DEFAULT_TABLE table=test-table [id=5ed37cba6cce45aca3a1dcbfc86654ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:45.179594 32127 tablet_service.cc:1467] Processing CreateTablet for tablet 9e950235fde54623bb477d0df660252f (DEFAULT_TABLE table=test-table [id=5ed37cba6cce45aca3a1dcbfc86654ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:45.179597 31986 tablet_service.cc:1467] Processing CreateTablet for tablet 9e950235fde54623bb477d0df660252f (DEFAULT_TABLE table=test-table [id=5ed37cba6cce45aca3a1dcbfc86654ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:45.179723 32127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e950235fde54623bb477d0df660252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:45.179725 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e950235fde54623bb477d0df660252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:45.179723 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e950235fde54623bb477d0df660252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:45.180933 32303 tablet_bootstrap.cc:492] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: Bootstrap starting.
I20260929 18:52:45.181216 32304 tablet_bootstrap.cc:492] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: Bootstrap starting.
I20260929 18:52:45.181268 32305 tablet_bootstrap.cc:492] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: Bootstrap starting.
I20260929 18:52:45.181465 32303 tablet_bootstrap.cc:654] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:45.181605 32304 tablet_bootstrap.cc:654] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:45.181670 32305 tablet_bootstrap.cc:654] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:45.182327 32303 tablet_bootstrap.cc:492] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: No bootstrap required, opened a new log
I20260929 18:52:45.182313 32304 tablet_bootstrap.cc:492] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: No bootstrap required, opened a new log
I20260929 18:52:45.182380 32304 ts_tablet_manager.cc:1397] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:45.182384 32303 ts_tablet_manager.cc:1397] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:45.182503 32305 tablet_bootstrap.cc:492] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: No bootstrap required, opened a new log
I20260929 18:52:45.182556 32305 ts_tablet_manager.cc:1397] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:45.182525 32304 raft_consensus.cc:348] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.182499 32303 raft_consensus.cc:348] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.182603 32304 raft_consensus.cc:374] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:45.182602 32303 raft_consensus.cc:374] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:45.182626 32304 raft_consensus.cc:729] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90fd353351744b38cbdc2f19bf67ef8, State: Initialized, Role: FOLLOWER
I20260929 18:52:45.182631 32303 raft_consensus.cc:729] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffefa730ca9f4c4086bac0c64756aaf0, State: Initialized, Role: FOLLOWER
I20260929 18:52:45.182690 32304 consensus_queue.cc:260] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.182693 32303 consensus_queue.cc:260] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.182654 32305 raft_consensus.cc:348] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.182744 32305 raft_consensus.cc:374] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:45.182777 32305 raft_consensus.cc:729] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4363ace3311843c09b62ff1363942478, State: Initialized, Role: FOLLOWER
I20260929 18:52:45.182861 32304 ts_tablet_manager.cc:1428] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:45.182847 32305 consensus_queue.cc:260] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.182870 32303 ts_tablet_manager.cc:1428] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:45.182946 32305 ts_tablet_manager.cc:1428] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:45.195056 32283 raft_consensus.cc:482] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:45.195144 32283 raft_consensus.cc:504] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.195335 32283 leader_election.cc:288] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:45.195520 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e950235fde54623bb477d0df660252f" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8" is_pre_election: true
I20260929 18:52:45.195523 31866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e950235fde54623bb477d0df660252f" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" is_pre_election: true
I20260929 18:52:45.195608 32147 raft_consensus.cc:2454] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 0.
I20260929 18:52:45.195608 31866 raft_consensus.cc:2454] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 0.
I20260929 18:52:45.195765 31939 leader_election.cc:302] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, a90fd353351744b38cbdc2f19bf67ef8; no voters:
I20260929 18:52:45.195884 32283 raft_consensus.cc:2781] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:45.195932 32283 raft_consensus.cc:482] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:45.195955 32283 raft_consensus.cc:3037] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:45.196729 32283 raft_consensus.cc:504] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.196868 32283 leader_election.cc:288] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 election: Requested vote from peers ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:45.197000 31866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e950235fde54623bb477d0df660252f" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffefa730ca9f4c4086bac0c64756aaf0"
I20260929 18:52:45.197021 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e950235fde54623bb477d0df660252f" candidate_uuid: "4363ace3311843c09b62ff1363942478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8"
I20260929 18:52:45.197103 32147 raft_consensus.cc:3037] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:45.197104 31866 raft_consensus.cc:3037] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:45.197685 32147 raft_consensus.cc:2454] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 1.
I20260929 18:52:45.197700 31866 raft_consensus.cc:2454] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4363ace3311843c09b62ff1363942478 in term 1.
I20260929 18:52:45.197852 31939 leader_election.cc:302] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, a90fd353351744b38cbdc2f19bf67ef8; no voters:
I20260929 18:52:45.197958 32283 raft_consensus.cc:2781] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:45.198052 32283 raft_consensus.cc:686] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 1 LEADER]: Becoming Leader. State: Replica: 4363ace3311843c09b62ff1363942478, State: Running, Role: LEADER
I20260929 18:52:45.198107 32283 consensus_queue.cc:237] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.198709 31708 catalog_manager.cc:5495] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4363ace3311843c09b62ff1363942478 (127.29.71.130). New cstate: current_term: 1 leader_uuid: "4363ace3311843c09b62ff1363942478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 18:52:45.207962 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:45.205414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.29.71.130:42411)
0929 18:52:45.205569 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:45.205572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:45.205579 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:52:45.205606 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:45.205607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:45.205615 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:45.205692 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:45.205844 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:45.206425 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:45.207220 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:45.207438 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:45.207538 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:45.207697 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:45.207737 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:45.207801 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:45.207859 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:52:45.207879 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260929 18:52:45.209995 32147 raft_consensus.cc:1261] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Refusing update from remote peer 4363ace3311843c09b62ff1363942478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:45.209990 31866 raft_consensus.cc:1261] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Refusing update from remote peer 4363ace3311843c09b62ff1363942478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:45.210211 32223 consensus_queue.cc:1035] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:45.210283 32283 consensus_queue.cc:1035] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:45.219866 31710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:45.219954 31710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:45.220546 31710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Sending DeleteTablet for 3 replicas of tablet 9e950235fde54623bb477d0df660252f
I20260929 18:52:45.220809 31986 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e950235fde54623bb477d0df660252f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:45 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:53454
I20260929 18:52:45.220820 31847 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e950235fde54623bb477d0df660252f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:45 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59854
I20260929 18:52:45.220867 32127 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e950235fde54623bb477d0df660252f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:45 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:42354
I20260929 18:52:45.220921 32292 tablet_replica.cc:330] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: stopping tablet replica
I20260929 18:52:45.220991 32292 raft_consensus.cc:2229] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:45.221011 32293 tablet_replica.cc:330] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: stopping tablet replica
I20260929 18:52:45.221042 32292 raft_consensus.cc:2258] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:45.221071 32293 raft_consensus.cc:2229] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:45.221119 32293 raft_consensus.cc:2258] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:45.221132 32294 tablet_replica.cc:330] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: stopping tablet replica
I20260929 18:52:45.221181 32294 raft_consensus.cc:2229] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:45.221128 31710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:52:45.221225 32294 raft_consensus.cc:2258] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:52:45.221253 31710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:45.221416 32293 ts_tablet_manager.cc:1905] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:45.221581 32292 ts_tablet_manager.cc:1905] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:45.223101 32292 ts_tablet_manager.cc:1918] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:45.223125 32293 ts_tablet_manager.cc:1918] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:45.223161 32292 log.cc:1196] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/9e950235fde54623bb477d0df660252f
I20260929 18:52:45.223181 32293 log.cc:1196] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/9e950235fde54623bb477d0df660252f
I20260929 18:52:45.223206 32294 ts_tablet_manager.cc:1905] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:45.223387 32292 ts_tablet_manager.cc:1939] T 9e950235fde54623bb477d0df660252f P ffefa730ca9f4c4086bac0c64756aaf0: Deleting consensus metadata
I20260929 18:52:45.223414 32293 ts_tablet_manager.cc:1939] T 9e950235fde54623bb477d0df660252f P a90fd353351744b38cbdc2f19bf67ef8: Deleting consensus metadata
I20260929 18:52:45.223740 31697 catalog_manager.cc:4841] TS ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565): tablet 9e950235fde54623bb477d0df660252f (table test-table [id=5ed37cba6cce45aca3a1dcbfc86654ce]) successfully deleted
I20260929 18:52:45.224155 31847 tablet_service.cc:1467] Processing CreateTablet for tablet 43c18fc07a064db1ba64677218e29712 (DEFAULT_TABLE table=test-table [id=247f4725e8244f55b60ae614fb48fafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:45.224321 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c18fc07a064db1ba64677218e29712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:45.224761 31695 catalog_manager.cc:4841] TS a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333): tablet 9e950235fde54623bb477d0df660252f (table test-table [id=5ed37cba6cce45aca3a1dcbfc86654ce]) successfully deleted
I20260929 18:52:45.224884 32294 ts_tablet_manager.cc:1918] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:52:45.224895 32127 tablet_service.cc:1467] Processing CreateTablet for tablet 43c18fc07a064db1ba64677218e29712 (DEFAULT_TABLE table=test-table [id=247f4725e8244f55b60ae614fb48fafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:45.224931 32294 log.cc:1196] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/9e950235fde54623bb477d0df660252f
I20260929 18:52:45.224982 32127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c18fc07a064db1ba64677218e29712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:45.225085 32294 ts_tablet_manager.cc:1939] T 9e950235fde54623bb477d0df660252f P 4363ace3311843c09b62ff1363942478: Deleting consensus metadata
I20260929 18:52:45.225329 32303 tablet_bootstrap.cc:492] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0: Bootstrap starting.
I20260929 18:52:45.225339 31986 tablet_service.cc:1467] Processing CreateTablet for tablet 43c18fc07a064db1ba64677218e29712 (DEFAULT_TABLE table=test-table [id=247f4725e8244f55b60ae614fb48fafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:45.225425 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c18fc07a064db1ba64677218e29712. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:45.225492 31694 catalog_manager.cc:4841] TS 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411): tablet 9e950235fde54623bb477d0df660252f (table test-table [id=5ed37cba6cce45aca3a1dcbfc86654ce]) successfully deleted
I20260929 18:52:45.225631 32303 tablet_bootstrap.cc:654] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:45.226141 32304 tablet_bootstrap.cc:492] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: Bootstrap starting.
I20260929 18:52:45.226151 32303 tablet_bootstrap.cc:492] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0: No bootstrap required, opened a new log
I20260929 18:52:45.226208 32303 ts_tablet_manager.cc:1397] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:52:45.226308 32303 raft_consensus.cc:348] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.226377 32303 raft_consensus.cc:374] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:45.226403 32303 raft_consensus.cc:729] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffefa730ca9f4c4086bac0c64756aaf0, State: Initialized, Role: FOLLOWER
I20260929 18:52:45.226464 32304 tablet_bootstrap.cc:654] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:45.226459 32303 consensus_queue.cc:260] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.226568 32303 ts_tablet_manager.cc:1428] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:45.226686 32305 tablet_bootstrap.cc:492] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: Bootstrap starting.
I20260929 18:52:45.226977 32305 tablet_bootstrap.cc:654] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:45.227531 32305 tablet_bootstrap.cc:492] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: No bootstrap required, opened a new log
I20260929 18:52:45.227535 32304 tablet_bootstrap.cc:492] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: No bootstrap required, opened a new log
I20260929 18:52:45.227602 32305 ts_tablet_manager.cc:1397] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:45.227595 32304 ts_tablet_manager.cc:1397] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:45.227706 32305 raft_consensus.cc:348] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.227747 32304 raft_consensus.cc:348] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.227787 32305 raft_consensus.cc:374] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:45.227800 32304 raft_consensus.cc:374] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:45.227829 32305 raft_consensus.cc:729] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4363ace3311843c09b62ff1363942478, State: Initialized, Role: FOLLOWER
I20260929 18:52:45.227849 32304 raft_consensus.cc:729] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90fd353351744b38cbdc2f19bf67ef8, State: Initialized, Role: FOLLOWER
I20260929 18:52:45.227896 32305 consensus_queue.cc:260] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.227898 32304 consensus_queue.cc:260] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.227989 32304 ts_tablet_manager.cc:1428] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:45.227993 32305 ts_tablet_manager.cc:1428] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:45.245263 32310 raft_consensus.cc:482] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:45.245404 32310 raft_consensus.cc:504] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.245568 32310 leader_election.cc:288] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:45.245832 32007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c18fc07a064db1ba64677218e29712" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4363ace3311843c09b62ff1363942478" is_pre_election: true
I20260929 18:52:45.245838 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c18fc07a064db1ba64677218e29712" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8" is_pre_election: true
I20260929 18:52:45.245926 32147 raft_consensus.cc:2454] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 0.
I20260929 18:52:45.245926 32007 raft_consensus.cc:2454] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 0.
I20260929 18:52:45.246070 31798 leader_election.cc:302] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, ffefa730ca9f4c4086bac0c64756aaf0; no voters:
I20260929 18:52:45.246209 32310 raft_consensus.cc:2781] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:45.246248 32310 raft_consensus.cc:482] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:45.246279 32310 raft_consensus.cc:3037] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:45.247037 32310 raft_consensus.cc:504] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.247187 32310 leader_election.cc:288] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 election: Requested vote from peers 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411), a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333)
I20260929 18:52:45.247340 32147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c18fc07a064db1ba64677218e29712" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90fd353351744b38cbdc2f19bf67ef8"
I20260929 18:52:45.247416 32147 raft_consensus.cc:3037] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:45.247396 32007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c18fc07a064db1ba64677218e29712" candidate_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4363ace3311843c09b62ff1363942478"
I20260929 18:52:45.247493 32007 raft_consensus.cc:3037] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:45.248083 32147 raft_consensus.cc:2454] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 1.
I20260929 18:52:45.248131 32007 raft_consensus.cc:2454] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffefa730ca9f4c4086bac0c64756aaf0 in term 1.
I20260929 18:52:45.248309 31798 leader_election.cc:302] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4363ace3311843c09b62ff1363942478, ffefa730ca9f4c4086bac0c64756aaf0; no voters:
I20260929 18:52:45.248447 32310 raft_consensus.cc:2781] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:45.248584 32310 raft_consensus.cc:686] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 LEADER]: Becoming Leader. State: Replica: ffefa730ca9f4c4086bac0c64756aaf0, State: Running, Role: LEADER
I20260929 18:52:45.248657 32310 consensus_queue.cc:237] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } }
I20260929 18:52:45.249274 31708 catalog_manager.cc:5495] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129). New cstate: current_term: 1 leader_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffefa730ca9f4c4086bac0c64756aaf0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260929 18:52:45.260725 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:45.258302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.29.71.129:33565)
0929 18:52:45.258479 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:45.258480 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:45.258486 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:52:45.258511 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:45.258511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:45.258519 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:45.258601 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:45.258710 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:45.259238 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:45.259989 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:45.260205 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:45.260287 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:45.260416 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:45.260454 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:45.260534 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:45.260599 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:52:45.260623 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260929 18:52:45.263264 32147 raft_consensus.cc:1261] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Refusing update from remote peer ffefa730ca9f4c4086bac0c64756aaf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:45.263312 32007 raft_consensus.cc:1261] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Refusing update from remote peer ffefa730ca9f4c4086bac0c64756aaf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:45.263489 32321 consensus_queue.cc:1035] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90fd353351744b38cbdc2f19bf67ef8" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:45.263593 32310 consensus_queue.cc:1035] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4363ace3311843c09b62ff1363942478" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:52:45.267974 32276 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:52:46.785498 31708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:46.785624 31708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:46.786542 31708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23c35e6b25ac40b7be591fc8c0a7b543: Sending DeleteTablet for 3 replicas of tablet 43c18fc07a064db1ba64677218e29712
I20260929 18:52:46.786813 31847 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c18fc07a064db1ba64677218e29712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:46 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59854
I20260929 18:52:46.786877 31986 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c18fc07a064db1ba64677218e29712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:46 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:53454
I20260929 18:52:46.787031 32127 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c18fc07a064db1ba64677218e29712 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:46 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:42354
I20260929 18:52:46.787043 32350 tablet_replica.cc:330] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0: stopping tablet replica
I20260929 18:52:46.787063 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 31782
I20260929 18:52:46.787110 32350 raft_consensus.cc:2229] T 43c18fc07a064db1ba64677218e29712 P ffefa730ca9f4c4086bac0c64756aaf0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:46.788048 32352 tablet_replica.cc:330] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: stopping tablet replica
I20260929 18:52:46.788130 32352 raft_consensus.cc:2229] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:46.788174 32352 raft_consensus.cc:2258] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:46.788293 32351 tablet_replica.cc:330] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: stopping tablet replica
I20260929 18:52:46.788379 32351 raft_consensus.cc:2229] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:46.788424 32351 raft_consensus.cc:2258] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:46.788444 32352 ts_tablet_manager.cc:1905] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:46.789753 32351 ts_tablet_manager.cc:1905] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:46.789948 32352 ts_tablet_manager.cc:1918] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:52:46.790005 32352 log.cc:1196] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/43c18fc07a064db1ba64677218e29712
I20260929 18:52:46.790241 32352 ts_tablet_manager.cc:1939] T 43c18fc07a064db1ba64677218e29712 P a90fd353351744b38cbdc2f19bf67ef8: Deleting consensus metadata
I20260929 18:52:46.790544 31695 catalog_manager.cc:4841] TS a90fd353351744b38cbdc2f19bf67ef8 (127.29.71.131:36333): tablet 43c18fc07a064db1ba64677218e29712 (table test-table [id=247f4725e8244f55b60ae614fb48fafb]) successfully deleted
I20260929 18:52:46.791379 32351 ts_tablet_manager.cc:1918] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:52:46.791427 32351 log.cc:1196] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/43c18fc07a064db1ba64677218e29712
I20260929 18:52:46.791641 32351 ts_tablet_manager.cc:1939] T 43c18fc07a064db1ba64677218e29712 P 4363ace3311843c09b62ff1363942478: Deleting consensus metadata
I20260929 18:52:46.792030 31694 catalog_manager.cc:4841] TS 4363ace3311843c09b62ff1363942478 (127.29.71.130:42411): tablet 43c18fc07a064db1ba64677218e29712 (table test-table [id=247f4725e8244f55b60ae614fb48fafb]) successfully deleted
W20260929 18:52:46.794484 31697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:33565 (error 108)
I20260929 18:52:46.795110 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 31922
I20260929 18:52:46.795174 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:46.794935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33565 (local address 127.0.0.1:59868)
0929 18:52:46.795079 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:46.795129 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:33565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260929 18:52:46.795836 31697 catalog_manager.cc:4584] TS ffefa730ca9f4c4086bac0c64756aaf0 (127.29.71.129:33565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43c18fc07a064db1ba64677218e29712: Network error: Client connection negotiation failed: client connection to 127.29.71.129:33565: connect: Connection refused (error 111)
I20260929 18:52:46.801430 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 32062
I20260929 18:52:46.805932 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 31678
2026-09-29T18:52:46Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5425 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:46 dist-test-slave-q2kh krb5kdc[32357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:46 dist-test-slave-q2kh krb5kdc[32357](info): set up 2 sockets
Sep 29 18:52:46 dist-test-slave-q2kh krb5kdc[32357](info): commencing operation
krb5kdc: starting...
W20260929 18:52:47.865994 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:47 dist-test-slave-q2kh krb5kdc[32357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:47.989192 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:37143
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:48.069602 32370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:48.069860 32370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:48.069900 32370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:48.071180 32370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:48.071260 32370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:48.071287 32370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:48.071318 32370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:48.071342 32370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:48.072784 32370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:37143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:48.073062 32370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:48.073292 32370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:48.078264 32378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:48.078248 32376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:48.078449 32370 server_base.cc:1034] running on GCE node
W20260929 18:52:48.078528 32375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:48.078766 32370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:48.079013 32370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:48.080170 32370 hybrid_clock.cc:648] HybridClock initialized: now 1790707968080155 us; error 30 us; skew 500 ppm
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:48.082826 32370 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:48.083276 32370 webserver.cc:460] Webserver started at http://127.29.71.190:33541/ using document root <none> and password file <none>
I20260929 18:52:48.083510 32370 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:48.083557 32370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:48.083714 32370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:48.084856 32370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "ce80cca97314454799bfe8fa7b557975"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "dc3823ada7e277b47b701e805599949c"
server_key_iv: "f869ba317558b0017890fe9713be9f22"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.085230 32370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "ce80cca97314454799bfe8fa7b557975"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "dc3823ada7e277b47b701e805599949c"
server_key_iv: "f869ba317558b0017890fe9713be9f22"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.086601 32370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:48.087294 32385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.087479 32370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:48.087582 32370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "ce80cca97314454799bfe8fa7b557975"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "dc3823ada7e277b47b701e805599949c"
server_key_iv: "f869ba317558b0017890fe9713be9f22"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.087668 32370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:48.102337 32370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:48.104158 32370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:48.104334 32370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:48.108263 32370 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:37143
I20260929 18:52:48.108299 32437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:37143 every 8 connection(s)
I20260929 18:52:48.108654 32370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:48.109534 32438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:48.111850 32438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975: Bootstrap starting.
I20260929 18:52:48.112424 32438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:48.112716 32438 log.cc:824] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:48.113399 32438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975: No bootstrap required, opened a new log
I20260929 18:52:48.114194 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 32370
I20260929 18:52:48.114289 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:48.114459 29982 external_mini_cluster.cc:1428] Setting key f61209878dc85d9e515a34aa7fb3beb6
I20260929 18:52:48.115675 32438 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce80cca97314454799bfe8fa7b557975" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 37143 } }
I20260929 18:52:48.115841 32438 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:48.115878 32438 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce80cca97314454799bfe8fa7b557975, State: Initialized, Role: FOLLOWER
I20260929 18:52:48.116061 32438 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce80cca97314454799bfe8fa7b557975" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 37143 } }
I20260929 18:52:48.116153 32438 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:48.116192 32438 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:48.116246 32438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:48.117306 32438 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce80cca97314454799bfe8fa7b557975" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 37143 } }
I20260929 18:52:48.117412 32438 leader_election.cc:302] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce80cca97314454799bfe8fa7b557975; no voters:
I20260929 18:52:48.117555 32438 leader_election.cc:288] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:48.117627 32443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:48.117775 32438 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:48.117784 32443 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [term 1 LEADER]: Becoming Leader. State: Replica: ce80cca97314454799bfe8fa7b557975, State: Running, Role: LEADER
I20260929 18:52:48.118010 32443 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce80cca97314454799bfe8fa7b557975" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 37143 } }
I20260929 18:52:48.118366 32445 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce80cca97314454799bfe8fa7b557975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce80cca97314454799bfe8fa7b557975" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 37143 } } }
I20260929 18:52:48.118363 32444 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce80cca97314454799bfe8fa7b557975. Latest consensus state: current_term: 1 leader_uuid: "ce80cca97314454799bfe8fa7b557975" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce80cca97314454799bfe8fa7b557975" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 37143 } } }
I20260929 18:52:48.118551 32445 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:48.118891 32444 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975 [sys.catalog]: This master's current role is: LEADER
W20260929 18:52:48.119777 32459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:52:48.120200 32459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:52:48.120280 32452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:48.120483 32452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:48.122573 32452 catalog_manager.cc:1347] Generated new cluster ID: bc0d9dcb1a80420bb683e99fd1bed7ea
I20260929 18:52:48.122627 32452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:52:48.126405 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:48.115065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.29.71.190:37143)
0929 18:52:48.115265 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:48.115275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:48.115309 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:52:48.115640 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:48.115649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:48.115711 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:48.115860 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:48.116680 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.118096 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.119308 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.119906 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.119997 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:48.122768 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:48.122771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:48.122772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:48.122782 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:48.124633 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.124985 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.124988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.124988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.125103 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.125275 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.125275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.125276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.125546 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:48.125678 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:48.126070 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 18:52:48.126216 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260929 18:52:48.130893 32452 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:48.131388 32452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:48.134719 32452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce80cca97314454799bfe8fa7b557975: Generated new TSK 0
I20260929 18:52:48.134883 32452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:52:48.166325 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:37143
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:48.247746 32464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:48.247954 32464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:48.247977 32464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:48.249166 32464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:48.249207 32464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:48.249233 32464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:48.250784 32464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:48.251036 32464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:48.251286 32464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:48.256259 32469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:48.256268 32470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:48.256268 32472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:48.256619 32464 server_base.cc:1034] running on GCE node
I20260929 18:52:48.256840 32464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:48.257097 32464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:48.258275 32464 hybrid_clock.cc:648] HybridClock initialized: now 1790707968258258 us; error 29 us; skew 500 ppm
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:48.261157 32464 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:48.261667 32464 webserver.cc:460] Webserver started at http://127.29.71.129:43785/ using document root <none> and password file <none>
I20260929 18:52:48.261874 32464 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:48.261920 32464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:48.262015 32464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:48.263168 32464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "142f42dbd83d40dbbe7fd3bf6e651558"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "23a678d86a5274ef1646cc181463a6ab"
server_key_iv: "4827b8a7b87a8b4ff09e6a039a90bd70"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.263499 32464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "142f42dbd83d40dbbe7fd3bf6e651558"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "23a678d86a5274ef1646cc181463a6ab"
server_key_iv: "4827b8a7b87a8b4ff09e6a039a90bd70"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.264834 32464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:48.265645 32479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.265899 32464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:48.265996 32464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "142f42dbd83d40dbbe7fd3bf6e651558"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "23a678d86a5274ef1646cc181463a6ab"
server_key_iv: "4827b8a7b87a8b4ff09e6a039a90bd70"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.266062 32464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:48.288894 32464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:48.291556 32464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:48.291718 32464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:48.291985 32464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:48.292331 32464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:48.292394 32464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.292430 32464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:48.292456 32464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.298041 32464 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:40379
I20260929 18:52:48.298141 32592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:40379 every 8 connection(s)
I20260929 18:52:48.298462 32464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:52:48.301231 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 32464
I20260929 18:52:48.301347 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:48.301514 29982 external_mini_cluster.cc:1428] Setting key 098c52f240785ec53c6ce6323e498c81
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:48.309267 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:48.299262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:33851 (local address 127.29.71.190:37143)
0929 18:52:48.299307 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:48.299309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:48.299498 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:52:48.300039 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:48.300040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:48.300059 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:48.300106 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:48.301084 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.301724 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.303274 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.303505 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.303581 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:48.305890 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:48.305891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:48.305892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:48.305903 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:48.307685 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.308193 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.308194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.308195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.308263 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.308558 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.308559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.308559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.308705 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:48.308783 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:48.309053 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 18:52:48.309152 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:52:48.309278 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:48.299222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:37143 (local address 127.29.71.129:33851)
0929 18:52:48.299358 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:48.299407 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:48.299920 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:48.300179 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:48.300183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:48.300766 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:48.300955 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:48.300961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.301845 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.301846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:48.303175 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:48.303178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.303607 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.303609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:48.303765 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:48.304345 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:48.304354 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:48.305777 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:48.307844 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:48.307851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:48.307853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:48.308100 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:48.308362 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:48.308363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:48.308363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:48.308474 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:48.308782 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:48.308786 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:48.308898 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 18:52:48.308997 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:52:48.309117 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260929 18:52:48.309863 32593 heartbeater.cc:347] Connected to a master server at 127.29.71.190:37143
I20260929 18:52:48.309970 32593 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
I20260929 18:52:48.310190 32593 heartbeater.cc:511] Master 127.29.71.190:37143 requested a full tablet report, sending...
I20260929 18:52:48.310695 32402 ts_manager.cc:194] Registered new tserver with Master: 142f42dbd83d40dbbe7fd3bf6e651558 (127.29.71.129:40379)
I20260929 18:52:48.311421 32402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:33851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:48.330372 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:37143
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:48.409195 32598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:48.409410 32598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:48.409435 32598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:48.410648 32598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:48.410694 32598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:48.410722 32598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:48.412103 32598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:48.412290 32598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:48.412497 32598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:48.417598 32604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:48.417613 32603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:48.417676 32598 server_base.cc:1034] running on GCE node
W20260929 18:52:48.417613 32606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:48.418264 32598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:48.418526 32598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:48.419694 32598 hybrid_clock.cc:648] HybridClock initialized: now 1790707968419679 us; error 56 us; skew 500 ppm
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:48.422116 32598 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:48.422724 32598 webserver.cc:460] Webserver started at http://127.29.71.130:32927/ using document root <none> and password file <none>
I20260929 18:52:48.422961 32598 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:48.423032 32598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:48.423182 32598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:48.424207 32598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "f34687b334b9408c8b1c94e78172a88a"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "31a06a969d3b8e713040159e1dc4f4c2"
server_key_iv: "5ee1cb3ad6aee873c7460ed88a5f2662"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.424580 32598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "f34687b334b9408c8b1c94e78172a88a"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "31a06a969d3b8e713040159e1dc4f4c2"
server_key_iv: "5ee1cb3ad6aee873c7460ed88a5f2662"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.425920 32598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 18:52:48.426681 32613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.426860 32598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.426960 32598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "f34687b334b9408c8b1c94e78172a88a"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "31a06a969d3b8e713040159e1dc4f4c2"
server_key_iv: "5ee1cb3ad6aee873c7460ed88a5f2662"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.427042 32598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:48.435331 32598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:48.437229 32598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:48.437394 32598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:48.437614 32598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:48.437911 32598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:48.437973 32598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.438014 32598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:48.438040 32598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.443929 32598 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:35945
I20260929 18:52:48.443966 32726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:35945 every 8 connection(s)
I20260929 18:52:48.444305 32598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:52:48.444749 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 32598
I20260929 18:52:48.444854 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:48.445058 29982 external_mini_cluster.cc:1428] Setting key 1b8a40bcb711a45b1a6a3fb437eedee8
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
I20260929 18:52:48.455569 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:48.445292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:34963 (local address 127.29.71.190:37143)
0929 18:52:48.445357 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:48.445360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:48.445586 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:52:48.446144 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:48.446146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:48.446174 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:48.446237 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:48.447425 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.447954 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.449624 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.449826 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.449886 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:48.452597 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:48.452599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:48.452599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:48.452609 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:48.454235 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.454633 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.454638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.454638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.454689 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.454967 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.454968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.454968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.455098 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:48.455171 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:48.455422 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 18:52:48.455500 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:52:48.455732 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:48.445250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:37143 (local address 127.29.71.130:34963)
0929 18:52:48.445427 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:48.445477 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:48.446039 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:48.446314 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:48.446317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:48.446970 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:48.447281 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:48.447289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.448061 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.448063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:48.449513 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:48.449517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.449932 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.449933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:48.450091 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:48.450676 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:48.450687 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:48.452469 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:48.454316 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:48.454322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:48.454324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:48.454547 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:48.454759 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:48.454760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:48.454760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:48.454875 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:48.455175 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:48.455179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:48.455296 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:52:48.455381 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:52:48.455510 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260929 18:52:48.456364 32727 heartbeater.cc:347] Connected to a master server at 127.29.71.190:37143
I20260929 18:52:48.456534 32727 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
I20260929 18:52:48.456748 32727 heartbeater.cc:511] Master 127.29.71.190:37143 requested a full tablet report, sending...
I20260929 18:52:48.457194 32402 ts_manager.cc:194] Registered new tserver with Master: f34687b334b9408c8b1c94e78172a88a (127.29.71.130:35945)
I20260929 18:52:48.457592 32402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:34963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:48.473749 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:37143
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:48.553576 32732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:48.553771 32732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:48.553793 32732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:48.554867 32732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:48.554901 32732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:48.554926 32732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:48.556226 32732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:48.556406 32732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:48.556602 32732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:48.561642 32738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:48.561642 32740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:48.561822 32737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:48.562014 32732 server_base.cc:1034] running on GCE node
I20260929 18:52:48.562244 32732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:48.562508 32732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:48.563658 32732 hybrid_clock.cc:648] HybridClock initialized: now 1790707968563630 us; error 41 us; skew 500 ppm
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:48.566387 32732 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:48.566839 32732 webserver.cc:460] Webserver started at http://127.29.71.131:33419/ using document root <none> and password file <none>
I20260929 18:52:48.567072 32732 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:48.567143 32732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:48.567273 32732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:48.568310 32732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "a33c9d96260a44f895927c27ce08913c"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "fbefd4d4b80d33ed02ec3f14a57cfb67"
server_key_iv: "dca104c190081021dfc85b97d45d5c56"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.568674 32732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "a33c9d96260a44f895927c27ce08913c"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "fbefd4d4b80d33ed02ec3f14a57cfb67"
server_key_iv: "dca104c190081021dfc85b97d45d5c56"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.570024 32732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 18:52:48.570757 32747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.570938 32732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.571036 32732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "a33c9d96260a44f895927c27ce08913c"
format_stamp: "Formatted at 2026-09-29 18:52:48 on dist-test-slave-q2kh"
server_key: "fbefd4d4b80d33ed02ec3f14a57cfb67"
server_key_iv: "dca104c190081021dfc85b97d45d5c56"
server_key_version: "encryptionkey@0"
I20260929 18:52:48.571123 32732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:48.604975 32732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:48.607407 32732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:48.607585 32732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:48.607841 32732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:48.608179 32732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:48.608237 32732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.608283 32732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:48.608312 32732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:48.614123 32732 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:37689
I20260929 18:52:48.614193 392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:37689 every 8 connection(s)
I20260929 18:52:48.614499 32732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:52:48.618887 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 32732
I20260929 18:52:48.619014 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:48.619246 29982 external_mini_cluster.cc:1428] Setting key d1c5fefe922719c728c6153e8f56d14d
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[32357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:48.623963 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:48.615224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:53121 (local address 127.29.71.190:37143)
0929 18:52:48.615319 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:48.615322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:48.615512 (+ 190us) server_negotiation.cc:407] Connection header received
0929 18:52:48.616042 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:48.616044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:48.616056 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:48.616113 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:48.617083 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.617654 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.618787 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.618986 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.619070 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:48.621473 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:48.621474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:48.621475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:48.621481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:48.622772 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.623177 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.623178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.623179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.623228 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:48.623442 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:48.623442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:48.623443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:48.623544 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:48.623594 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:48.623839 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:52:48.623902 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:52:48.624162 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:48.615185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:37143 (local address 127.29.71.131:53121)
0929 18:52:48.615379 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:48.615417 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:48.615933 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:48.616189 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:48.616198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:48.616776 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:48.616976 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:48.616981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.617746 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.617747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:48.618696 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:48.618699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:48.619094 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:48.619096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:48.619256 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:48.619883 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:48.619894 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:48.621372 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:48.622845 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:48.622850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:48.622852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:48.623109 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:48.623288 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:48.623289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:48.623289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:48.623383 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:48.623601 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:48.623604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:48.623723 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 18:52:48.623808 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:52:48.623920 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260929 18:52:48.624673 393 heartbeater.cc:347] Connected to a master server at 127.29.71.190:37143
I20260929 18:52:48.624778 393 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:48.624965 393 heartbeater.cc:511] Master 127.29.71.190:37143 requested a full tablet report, sending...
I20260929 18:52:48.625350 32402 ts_manager.cc:194] Registered new tserver with Master: a33c9d96260a44f895927c27ce08913c (127.29.71.131:37689)
I20260929 18:52:48.625715 32402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:53121
I20260929 18:52:48.631105 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:52:48.641983 32442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:52:48.641076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.29.71.190:37143)
0929 18:52:48.641167 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:48.641170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:48.641198 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:52:48.641266 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:48.641267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:48.641276 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:48.641322 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:48.641928 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45126: BlockingRecv error: recv got EOF from 127.0.0.1:45126 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:52:48.642833 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 32464
I20260929 18:52:48.647143 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 32598
I20260929 18:52:48.651057 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 32732
I20260929 18:52:48.655306 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 32370
2026-09-29T18:52:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1846 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[405](info): set up 2 sockets
Sep 29 18:52:48 dist-test-slave-q2kh krb5kdc[405](info): commencing operation
krb5kdc: starting...
W20260929 18:52:49.690196 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:49 dist-test-slave-q2kh krb5kdc[405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:49.808936 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42037
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:49.908825 418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:49.909080 418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:49.909119 418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:49.910359 418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:49.910425 418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:49.910462 418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:49.910490 418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:49.910511 418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:49.912027 418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:42037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:49.912315 418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:49.912552 418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:49.917863 426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:49.917868 423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:49.918069 418 server_base.cc:1034] running on GCE node
W20260929 18:52:49.917863 424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:49.918399 418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:49.918707 418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:49.919884 418 hybrid_clock.cc:648] HybridClock initialized: now 1790707969919853 us; error 41 us; skew 500 ppm
Sep 29 18:52:49 dist-test-slave-q2kh krb5kdc[405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707969, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:49.922420 418 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:49.922886 418 webserver.cc:460] Webserver started at http://127.29.71.190:40473/ using document root <none> and password file <none>
I20260929 18:52:49.923091 418 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:49.923132 418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:49.923221 418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:49.924212 418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "700c423457a647d99c8c59aaa9e5ce9e"
format_stamp: "Formatted at 2026-09-29 18:52:49 on dist-test-slave-q2kh"
server_key: "b475ad37f67b0b00f58607931df7ba63"
server_key_iv: "abe1913057d7ebe19b857409b512605b"
server_key_version: "encryptionkey@0"
I20260929 18:52:49.924530 418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "700c423457a647d99c8c59aaa9e5ce9e"
format_stamp: "Formatted at 2026-09-29 18:52:49 on dist-test-slave-q2kh"
server_key: "b475ad37f67b0b00f58607931df7ba63"
server_key_iv: "abe1913057d7ebe19b857409b512605b"
server_key_version: "encryptionkey@0"
I20260929 18:52:49.925765 418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:52:49.926524 433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:49.926710 418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:49.926791 418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "700c423457a647d99c8c59aaa9e5ce9e"
format_stamp: "Formatted at 2026-09-29 18:52:49 on dist-test-slave-q2kh"
server_key: "b475ad37f67b0b00f58607931df7ba63"
server_key_iv: "abe1913057d7ebe19b857409b512605b"
server_key_version: "encryptionkey@0"
I20260929 18:52:49.926843 418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:49.936195 418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:49.938462 418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:49.938575 418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:49.942202 418 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:42037
I20260929 18:52:49.942235 485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:42037 every 8 connection(s)
I20260929 18:52:49.942742 418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:49.943423 486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:49.943503 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 418
I20260929 18:52:49.943620 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:49.943804 29982 external_mini_cluster.cc:1428] Setting key 9e5f871ddc51212adfac2db937dd9049
I20260929 18:52:49.946889 486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e: Bootstrap starting.
I20260929 18:52:49.947580 486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:49.947876 486 log.cc:824] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:49.948567 486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e: No bootstrap required, opened a new log
Sep 29 18:52:49 dist-test-slave-q2kh krb5kdc[405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:49.950579 486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c423457a647d99c8c59aaa9e5ce9e" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42037 } }
I20260929 18:52:49.950733 486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:49.950769 486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700c423457a647d99c8c59aaa9e5ce9e, State: Initialized, Role: FOLLOWER
I20260929 18:52:49.950901 486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c423457a647d99c8c59aaa9e5ce9e" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42037 } }
I20260929 18:52:49.950994 486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:49.951026 486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:49.951076 486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:49.951596 486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c423457a647d99c8c59aaa9e5ce9e" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42037 } }
I20260929 18:52:49.951730 486 leader_election.cc:302] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700c423457a647d99c8c59aaa9e5ce9e; no voters:
I20260929 18:52:49.951936 486 leader_election.cc:288] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:49.951974 491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:49.952078 491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [term 1 LEADER]: Becoming Leader. State: Replica: 700c423457a647d99c8c59aaa9e5ce9e, State: Running, Role: LEADER
I20260929 18:52:49.952157 491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c423457a647d99c8c59aaa9e5ce9e" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42037 } }
I20260929 18:52:49.952188 486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:49.952457 491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700c423457a647d99c8c59aaa9e5ce9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c423457a647d99c8c59aaa9e5ce9e" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42037 } } }
I20260929 18:52:49.952529 491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:49.952651 491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700c423457a647d99c8c59aaa9e5ce9e. Latest consensus state: current_term: 1 leader_uuid: "700c423457a647d99c8c59aaa9e5ce9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c423457a647d99c8c59aaa9e5ce9e" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42037 } } }
I20260929 18:52:49.952719 491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e [sys.catalog]: This master's current role is: LEADER
W20260929 18:52:49.953913 506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:52:49.953977 506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:52:49.954069 496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:49.954308 496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:49.955030 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:49.944346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.29.71.190:42037)
0929 18:52:49.944585 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:49.944595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:49.944625 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:52:49.945148 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:49.945158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:49.945225 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:49.945378 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:49.945993 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:49.947357 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:49.948187 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:49.948581 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:49.948704 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:49.951264 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:49.951267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:49.951267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:49.951276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:49.953221 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:49.953679 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:49.953682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:49.953682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:49.953872 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:49.954081 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:49.954082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:49.954082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:49.954411 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:49.954566 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:49.954693 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 18:52:49.954831 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":65,"threads_started":1}
I20260929 18:52:49.955889 496 catalog_manager.cc:1347] Generated new cluster ID: 9ed1a7a09c42454388dbd667d499fbbe
I20260929 18:52:49.955940 496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:52:49.956033 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:52:49.972445 496 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:49.972554 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:49.968218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.29.71.190:42037)
0929 18:52:49.968262 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:49.968264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:49.968413 (+ 149us) server_negotiation.cc:407] Connection header received
0929 18:52:49.968492 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:49.968493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:49.968506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:49.968565 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:49.969004 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:49.969745 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:49.970859 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:49.971155 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:49.971225 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:49.972002 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:49.972003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:49.972003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:49.972217 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:49.972233 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:49.972240 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:49.972324 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:49.972432 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 18:52:49.972471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:52:49.973148 496 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:49.979555 496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 700c423457a647d99c8c59aaa9e5ce9e: Generated new TSK 0
I20260929 18:52:49.979768 496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:52:49.985767 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 418
2026-09-29T18:52:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1328 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:50 dist-test-slave-q2kh krb5kdc[519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:50 dist-test-slave-q2kh krb5kdc[519](info): set up 2 sockets
Sep 29 18:52:50 dist-test-slave-q2kh krb5kdc[519](info): commencing operation
krb5kdc: starting...
W20260929 18:52:51.030234 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:51.149812 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33861
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:51.232759 532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:51.233001 532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:51.233037 532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:51.234375 532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:51.234452 532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:51.234477 532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:51.234509 532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:51.234530 532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:51.236029 532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:51.236307 532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:51.236568 532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:52:51.241400 532 server_base.cc:1034] running on GCE node
W20260929 18:52:51.241348 538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:51.241354 540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:51.241354 537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:51.241828 532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:51.242132 532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:51.243332 532 hybrid_clock.cc:648] HybridClock initialized: now 1790707971243321 us; error 27 us; skew 500 ppm
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:51.253427 532 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:51.253940 532 webserver.cc:460] Webserver started at http://127.29.71.190:41459/ using document root <none> and password file <none>
I20260929 18:52:51.254247 532 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:51.254324 532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:51.254488 532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:51.255610 532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "195e7d3434e94a4380e850a5a4a80567"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "a389ffcc511dd094101a5522a4f0f31c"
server_key_iv: "296cdf09c5e37f41a30db556345cc275"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.256004 532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "195e7d3434e94a4380e850a5a4a80567"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "a389ffcc511dd094101a5522a4f0f31c"
server_key_iv: "296cdf09c5e37f41a30db556345cc275"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.257386 532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:51.258144 547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.258404 532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:51.258497 532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "195e7d3434e94a4380e850a5a4a80567"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "a389ffcc511dd094101a5522a4f0f31c"
server_key_iv: "296cdf09c5e37f41a30db556345cc275"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.258592 532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:51.266669 532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:51.268978 532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:51.269124 532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:51.273087 532 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33861
I20260929 18:52:51.273120 599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33861 every 8 connection(s)
I20260929 18:52:51.273497 532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:51.274552 600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:51.275501 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 532
I20260929 18:52:51.275615 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:51.275794 29982 external_mini_cluster.cc:1428] Setting key 89a3d5e67b37fabe3a307f088edad936
I20260929 18:52:51.278100 600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567: Bootstrap starting.
I20260929 18:52:51.278753 600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:51.279071 600 log.cc:824] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:51.279830 600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567: No bootstrap required, opened a new log
I20260929 18:52:51.282418 600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e7d3434e94a4380e850a5a4a80567" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33861 } }
I20260929 18:52:51.282552 600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:51.282572 600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195e7d3434e94a4380e850a5a4a80567, State: Initialized, Role: FOLLOWER
I20260929 18:52:51.282727 600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e7d3434e94a4380e850a5a4a80567" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33861 } }
I20260929 18:52:51.282800 600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:51.282855 600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:51.282905 600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:51.283609 600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e7d3434e94a4380e850a5a4a80567" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33861 } }
I20260929 18:52:51.283747 600 leader_election.cc:302] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 195e7d3434e94a4380e850a5a4a80567; no voters:
I20260929 18:52:51.283932 600 leader_election.cc:288] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:51.284017 605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:51.284224 600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:51.284231 605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [term 1 LEADER]: Becoming Leader. State: Replica: 195e7d3434e94a4380e850a5a4a80567, State: Running, Role: LEADER
I20260929 18:52:51.284358 605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e7d3434e94a4380e850a5a4a80567" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33861 } }
I20260929 18:52:51.284818 606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "195e7d3434e94a4380e850a5a4a80567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e7d3434e94a4380e850a5a4a80567" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33861 } } }
I20260929 18:52:51.284951 606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:51.285220 614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:51.285776 614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:51.286044 607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 195e7d3434e94a4380e850a5a4a80567. Latest consensus state: current_term: 1 leader_uuid: "195e7d3434e94a4380e850a5a4a80567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195e7d3434e94a4380e850a5a4a80567" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33861 } } }
I20260929 18:52:51.286459 607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:51.287500 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.276341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.29.71.190:33861)
0929 18:52:51.276539 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:51.276552 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:51.276577 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:52:51.277004 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:51.277010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:51.277056 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:51.277169 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:51.278005 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.279007 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.280034 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.280427 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.280506 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.283423 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:51.283427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:51.283427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:51.283437 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:51.285690 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.286126 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.286129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.286129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.286256 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.286510 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.286511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.286511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.286807 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:51.286957 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:51.287140 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:52:51.287284 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260929 18:52:51.287770 614 catalog_manager.cc:1347] Generated new cluster ID: 80618b570d6c46b69f9a56034bc0863d
I20260929 18:52:51.287817 614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:52:51.291858 614 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:51.292343 614 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:51.294049 614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 195e7d3434e94a4380e850a5a4a80567: Generated new TSK 0
I20260929 18:52:51.294250 614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:52:51.324791 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33861
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:51.405365 626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:51.405604 626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:51.405639 626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:51.406804 626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:51.406875 626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:51.406919 626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:51.408322 626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:51.408586 626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:51.408811 626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:51.414172 631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:51.414240 634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:51.414192 626 server_base.cc:1034] running on GCE node
W20260929 18:52:51.414177 632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:51.414670 626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:51.414911 626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:51.416102 626 hybrid_clock.cc:648] HybridClock initialized: now 1790707971416074 us; error 38 us; skew 500 ppm
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:51.418622 626 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:51.419080 626 webserver.cc:460] Webserver started at http://127.29.71.129:43315/ using document root <none> and password file <none>
I20260929 18:52:51.419260 626 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:51.419301 626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:51.419384 626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:51.420432 626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "8a540fb22e8f4e6e8df0367bb423297c"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "29a976d695fdb0ed56c1364ac401edaf"
server_key_iv: "2306346092572f878bac7dc34b6a5d03"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.420728 626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "8a540fb22e8f4e6e8df0367bb423297c"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "29a976d695fdb0ed56c1364ac401edaf"
server_key_iv: "2306346092572f878bac7dc34b6a5d03"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.421955 626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:52:51.422704 641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.422883 626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.422971 626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "8a540fb22e8f4e6e8df0367bb423297c"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "29a976d695fdb0ed56c1364ac401edaf"
server_key_iv: "2306346092572f878bac7dc34b6a5d03"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.423072 626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:51.443478 626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:51.445396 626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:51.445551 626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:51.445822 626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:51.446146 626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:51.446272 626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.446316 626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:51.446343 626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.452085 626 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:36389
I20260929 18:52:51.452188 754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:36389 every 8 connection(s)
I20260929 18:52:51.452490 626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:51.460186 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 626
I20260929 18:52:51.460299 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:51.460503 29982 external_mini_cluster.cc:1428] Setting key 03835cfcbfd79ac77ceb1c60ee2bc785
I20260929 18:52:51.461547 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.453249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:40193 (local address 127.29.71.190:33861)
0929 18:52:51.453298 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:51.453301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:51.453511 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:52:51.453974 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:51.453975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:51.453985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:51.454036 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:51.455007 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.455497 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.456449 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.456659 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.456710 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.458724 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:51.458725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:51.458725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:51.458731 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:51.459895 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.460424 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.460425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.460425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.460479 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.460771 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.460772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.460772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.460934 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:51.461003 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:51.461328 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 18:52:51.461434 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:52:51.461710 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.453232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33861 (local address 127.29.71.129:40193)
0929 18:52:51.453384 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:51.453420 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:51.453871 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:51.454087 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:51.454091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:51.454719 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:51.454912 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:51.454926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.455579 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.455580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:51.456368 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:51.456370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.456733 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.456734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:51.456829 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.457266 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:51.457274 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:51.458642 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:51.460014 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:51.460023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:51.460025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:51.460329 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:51.460552 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:51.460553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:51.460553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:51.460690 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:51.460999 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:51.461004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:51.461142 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:52:51.461298 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 18:52:51.461494 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260929 18:52:51.462191 755 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33861
I20260929 18:52:51.462304 755 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:51.462564 755 heartbeater.cc:511] Master 127.29.71.190:33861 requested a full tablet report, sending...
I20260929 18:52:51.463111 564 ts_manager.cc:194] Registered new tserver with Master: 8a540fb22e8f4e6e8df0367bb423297c (127.29.71.129:36389)
I20260929 18:52:51.463760 564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:40193
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:51.485658 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33861
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:51.566030 760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:51.566325 760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:51.566367 760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:51.567562 760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:51.567642 760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:51.567687 760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:51.569136 760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:51.569399 760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:51.569636 760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:51.574839 765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:51.574944 768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:51.574862 766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:51.575317 760 server_base.cc:1034] running on GCE node
I20260929 18:52:51.575521 760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:51.575803 760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:51.577030 760 hybrid_clock.cc:648] HybridClock initialized: now 1790707971577008 us; error 43 us; skew 500 ppm
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:51.579450 760 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:51.580003 760 webserver.cc:460] Webserver started at http://127.29.71.130:40757/ using document root <none> and password file <none>
I20260929 18:52:51.580245 760 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:51.580289 760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:51.580454 760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:51.581481 760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "90ce0dad2d3c4131aab084762185b690"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "d30aa7857f0127cef1a2b0d9520ae132"
server_key_iv: "b55ace79b85aff5bcfea17e683411a4c"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.581837 760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "90ce0dad2d3c4131aab084762185b690"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "d30aa7857f0127cef1a2b0d9520ae132"
server_key_iv: "b55ace79b85aff5bcfea17e683411a4c"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.583213 760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:51.583984 775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.584163 760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:51.584251 760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "90ce0dad2d3c4131aab084762185b690"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "d30aa7857f0127cef1a2b0d9520ae132"
server_key_iv: "b55ace79b85aff5bcfea17e683411a4c"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.584369 760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:51.604709 760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:51.607110 760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:51.607280 760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:51.607517 760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:51.607824 760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:51.607880 760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.607930 760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:51.607975 760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.613538 760 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:38661
I20260929 18:52:51.613588 888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:38661 every 8 connection(s)
I20260929 18:52:51.613951 760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:51.620451 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 760
I20260929 18:52:51.620566 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:51.620786 29982 external_mini_cluster.cc:1428] Setting key f9208daf552b0de4db889af37820cb18
I20260929 18:52:51.623339 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.614743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:43021 (local address 127.29.71.190:33861)
0929 18:52:51.614833 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:51.614836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:51.615013 (+ 177us) server_negotiation.cc:407] Connection header received
0929 18:52:51.615474 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:51.615475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:51.615486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:51.615538 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:51.616423 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.616915 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.617879 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.618070 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.618124 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.620276 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:51.620300 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:51.620301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:51.620307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:51.621857 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.622373 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.622375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.622375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.622438 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.622695 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.622696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.622697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.622814 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:51.622893 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:51.623168 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:52:51.623245 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:52:51.623569 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.614696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33861 (local address 127.29.71.130:43021)
0929 18:52:51.614882 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:51.614919 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:51.615378 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:51.615592 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:51.615595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:51.616151 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:51.616340 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:51.616345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.616994 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.616995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:51.617797 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:51.617799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.618147 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.618147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:51.618317 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.618771 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:51.618779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:51.620168 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:51.621982 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:51.621989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:51.621991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:51.622284 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:51.622515 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:51.622516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:51.622516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:51.622626 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:51.622883 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:51.622889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:51.623034 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 18:52:51.623149 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:52:51.623310 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260929 18:52:51.624039 889 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33861
I20260929 18:52:51.624150 889 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:51.624328 889 heartbeater.cc:511] Master 127.29.71.190:33861 requested a full tablet report, sending...
I20260929 18:52:51.624733 564 ts_manager.cc:194] Registered new tserver with Master: 90ce0dad2d3c4131aab084762185b690 (127.29.71.130:38661)
I20260929 18:52:51.625288 564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:43021
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:51.648784 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33861
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:51.729250 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:51.729457 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:51.729480 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:51.730598 894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:51.730633 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:51.730657 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:51.732003 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:51.732188 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:51.732378 894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:52:51.738003 894 server_base.cc:1034] running on GCE node
W20260929 18:52:51.738003 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:51.738032 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:51.738070 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:51.738477 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:51.738730 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:51.739876 894 hybrid_clock.cc:648] HybridClock initialized: now 1790707971739856 us; error 38 us; skew 500 ppm
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:51.742393 894 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:51.742818 894 webserver.cc:460] Webserver started at http://127.29.71.131:42291/ using document root <none> and password file <none>
I20260929 18:52:51.743065 894 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:51.743137 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:51.743285 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:51.744375 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "d3d89536961f4617b42f003a506394ad"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "bf4af7b7c818f4581fe3b5235eab36b2"
server_key_iv: "9a97ccc506b0d8d9cb8e343e57bf9a03"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.744742 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d89536961f4617b42f003a506394ad"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "bf4af7b7c818f4581fe3b5235eab36b2"
server_key_iv: "9a97ccc506b0d8d9cb8e343e57bf9a03"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.746069 894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:51.746892 909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.747084 894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:51.747186 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "d3d89536961f4617b42f003a506394ad"
format_stamp: "Formatted at 2026-09-29 18:52:51 on dist-test-slave-q2kh"
server_key: "bf4af7b7c818f4581fe3b5235eab36b2"
server_key_iv: "9a97ccc506b0d8d9cb8e343e57bf9a03"
server_key_version: "encryptionkey@0"
I20260929 18:52:51.747267 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:51.758683 894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:51.760564 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:51.760783 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:51.761023 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:51.761327 894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:51.761386 894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.761425 894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:51.761454 894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:51.767225 894 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:38019
I20260929 18:52:51.767269 1022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:38019 every 8 connection(s)
I20260929 18:52:51.767583 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:51.774111 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 894
I20260929 18:52:51.774238 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:51.774454 29982 external_mini_cluster.cc:1428] Setting key 9560dd9de232de7235c99f0974811c98
I20260929 18:52:51.776890 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.768282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:39057 (local address 127.29.71.190:33861)
0929 18:52:51.768353 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:51.768355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:51.768566 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:52:51.769041 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:51.769042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:51.769052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:51.769111 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:51.770087 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.770657 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.771661 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.771846 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.771901 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.773896 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:51.773898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:51.773898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:51.773905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:51.775681 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.776059 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.776060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.776060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.776118 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.776335 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.776336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.776336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.776444 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:51.776507 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:51.776762 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 18:52:51.776827 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:52:51.777017 1025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.768256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33861 (local address 127.29.71.131:39057)
0929 18:52:51.768433 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:51.768476 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:51.768957 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:51.769198 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:51.769202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:51.769778 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:51.769977 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:51.769981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.770746 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.770747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:51.771572 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:51.771574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.771929 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.771930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:51.772038 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.772503 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:51.772511 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:51.773814 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:51.775750 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:51.775755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:51.775757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:51.775987 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:51.776183 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:51.776184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:51.776184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:51.776276 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:51.776498 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:51.776500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:51.776619 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 18:52:51.776697 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:52:51.776809 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260929 18:52:51.777444 1023 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33861
I20260929 18:52:51.777554 1023 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:51.777726 1023 heartbeater.cc:511] Master 127.29.71.190:33861 requested a full tablet report, sending...
I20260929 18:52:51.778105 564 ts_manager.cc:194] Registered new tserver with Master: d3d89536961f4617b42f003a506394ad (127.29.71.131:38019)
I20260929 18:52:51.778501 564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:39057
I20260929 18:52:51.786515 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707971, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:51.811152 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:51.803195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.29.71.190:33861)
0929 18:52:51.803297 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:51.803299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:51.803401 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:52:51.803454 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:51.803455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:51.803464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:51.803522 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:51.804227 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.804737 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.805482 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:51.805667 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:51.805724 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:51.808175 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:51.808177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:51.808177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:51.808185 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:51.809507 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.809956 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.809958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.809958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.810014 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:51.810259 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:51.810260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:51.810260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:51.810480 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:51.810587 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:51.810891 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 18:52:51.811032 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 18:52:51.811444 564 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49336
I20260929 18:52:51.811921 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 626
I20260929 18:52:51.816547 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 760
I20260929 18:52:51.821228 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 894
I20260929 18:52:51.825932 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 532
2026-09-29T18:52:51Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1842 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[1035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[1035](info): set up 2 sockets
Sep 29 18:52:51 dist-test-slave-q2kh krb5kdc[1035](info): commencing operation
krb5kdc: starting...
W20260929 18:52:52.939360 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 18:52:53.103773 29982 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.164s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 18:52:53.346298 29982 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.242s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 18:52:53.581849 29982 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.235s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/test-user.keytab.
W20260929 18:52:54.582665 29982 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.001s user 0.000s sys 0.002s
Sep 29 18:52:54 dist-test-slave-q2kh krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
W20260929 18:52:55.872542 29982 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.71.190: real 1.268s user 0.001s sys 0.002s
I20260929 18:52:55.872913 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:36577
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:55.953872 1048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:55.954097 1048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:55.954123 1048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:55.955322 1048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:55.955372 1048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:55.955389 1048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:55.955402 1048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:55.955459 1048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:55.957053 1048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:36577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:55.957335 1048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:55.957579 1048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:55.962548 1056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:55.962558 1053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:55.962569 1054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:55.962775 1048 server_base.cc:1034] running on GCE node
I20260929 18:52:55.963050 1048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:55.963315 1048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:55.964455 1048 hybrid_clock.cc:648] HybridClock initialized: now 1790707975964443 us; error 27 us; skew 500 ppm
Sep 29 18:52:55 dist-test-slave-q2kh krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:55.967020 1048 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:55.967474 1048 webserver.cc:460] Webserver started at http://127.29.71.190:45289/ using document root <none> and password file <none>
I20260929 18:52:55.967703 1048 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:55.967746 1048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:55.967906 1048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:55.968978 1048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8"
format_stamp: "Formatted at 2026-09-29 18:52:55 on dist-test-slave-q2kh"
server_key: "28eaca85d3bf365150b49784c01d6426"
server_key_iv: "5c3631a739fa0d5e876fb7e3a3ba3535"
server_key_version: "encryptionkey@0"
I20260929 18:52:55.969327 1048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8"
format_stamp: "Formatted at 2026-09-29 18:52:55 on dist-test-slave-q2kh"
server_key: "28eaca85d3bf365150b49784c01d6426"
server_key_iv: "5c3631a739fa0d5e876fb7e3a3ba3535"
server_key_version: "encryptionkey@0"
I20260929 18:52:55.970796 1048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:55.971948 1063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:55.972153 1048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:55.972256 1048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8"
format_stamp: "Formatted at 2026-09-29 18:52:55 on dist-test-slave-q2kh"
server_key: "28eaca85d3bf365150b49784c01d6426"
server_key_iv: "5c3631a739fa0d5e876fb7e3a3ba3535"
server_key_version: "encryptionkey@0"
I20260929 18:52:55.972342 1048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:55.979249 1048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:55.981511 1048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:55.981673 1048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:55.985818 1048 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:36577
I20260929 18:52:55.985849 1115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:36577 every 8 connection(s)
I20260929 18:52:55.986433 1048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:55.987102 1116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:55.987421 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1048
I20260929 18:52:55.987541 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:55.987732 29982 external_mini_cluster.cc:1428] Setting key 02c0e0aff9951c7b7a9ebdaeea374e0c
I20260929 18:52:55.989635 1116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8: Bootstrap starting.
I20260929 18:52:55.990278 1116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:55.990558 1116 log.cc:824] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:55.991204 1116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8: No bootstrap required, opened a new log
I20260929 18:52:55.993772 1116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36577 } }
I20260929 18:52:55.993957 1116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:55.993991 1116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee70fbe90544c7aa96d21d6a9a7dbb8, State: Initialized, Role: FOLLOWER
Sep 29 18:52:55 dist-test-slave-q2kh krb5kdc[1035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:55.994122 1116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36577 } }
I20260929 18:52:55.994235 1116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:55.994277 1116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:55.994350 1116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:55.997753 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:55.988269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.29.71.190:36577)
0929 18:52:55.988451 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:55.988458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:55.988483 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:52:55.988885 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:55.988892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:55.988950 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:55.989097 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:55.989712 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:55.990800 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:55.991603 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:55.992034 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:55.992171 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:55.994646 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:55.994650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:55.994651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:55.994662 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:55.996275 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:55.996674 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:55.996677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:55.996677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:55.996763 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:55.996971 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:55.996972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:55.996973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:55.997200 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:55.997346 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:55.997460 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 18:52:55.997594 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260929 18:52:56.066897 1116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36577 } }
I20260929 18:52:56.067111 1116 leader_election.cc:302] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ee70fbe90544c7aa96d21d6a9a7dbb8; no voters:
I20260929 18:52:56.067349 1116 leader_election.cc:288] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:56.067446 1121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:56.067656 1116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:56.067660 1121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [term 1 LEADER]: Becoming Leader. State: Replica: 2ee70fbe90544c7aa96d21d6a9a7dbb8, State: Running, Role: LEADER
I20260929 18:52:56.067791 1121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36577 } }
I20260929 18:52:56.068226 1123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ee70fbe90544c7aa96d21d6a9a7dbb8. Latest consensus state: current_term: 1 leader_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36577 } } }
I20260929 18:52:56.068321 1123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:56.068507 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee70fbe90544c7aa96d21d6a9a7dbb8" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36577 } } }
I20260929 18:52:56.068598 1130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:56.068847 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:56.069273 1130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:56.071388 1130 catalog_manager.cc:1347] Generated new cluster ID: fbc8e3735188413382c32a9ead0992e2
I20260929 18:52:56.071437 1130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:52:56.088742 1130 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:56.089356 1130 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:56.094707 1130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ee70fbe90544c7aa96d21d6a9a7dbb8: Generated new TSK 0
I20260929 18:52:56.094897 1130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
W20260929 18:52:56.524753 29982 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.71.129: real 0.427s user 0.000s sys 0.003s
I20260929 18:52:56.525132 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36577
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:56.607837 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:56.608032 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:56.608055 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:56.609117 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:56.609148 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:56.609174 1142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:56.610725 1142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:56.610912 1142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:56.611095 1142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:56.616008 1147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:56.616034 1148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:56.615986 1150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:56.616052 1142 server_base.cc:1034] running on GCE node
I20260929 18:52:56.616577 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:56.616822 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:56.617988 1142 hybrid_clock.cc:648] HybridClock initialized: now 1790707976617990 us; error 69 us; skew 500 ppm
Sep 29 18:52:56 dist-test-slave-q2kh krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:56.628885 1142 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:56.629346 1142 webserver.cc:460] Webserver started at http://127.29.71.129:41127/ using document root <none> and password file <none>
I20260929 18:52:56.629602 1142 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:56.629670 1142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:56.629814 1142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:56.630954 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "2fbf8f760b454d039dddb2ea688eeb8c"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "69b9002865159c3de62a3fb1dad7047c"
server_key_iv: "de9bd0d6ccd00dfdb5068512a8c20232"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.631306 1142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "2fbf8f760b454d039dddb2ea688eeb8c"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "69b9002865159c3de62a3fb1dad7047c"
server_key_iv: "de9bd0d6ccd00dfdb5068512a8c20232"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.632633 1142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:52:56.633344 1157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.633517 1142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:56.633615 1142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "2fbf8f760b454d039dddb2ea688eeb8c"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "69b9002865159c3de62a3fb1dad7047c"
server_key_iv: "de9bd0d6ccd00dfdb5068512a8c20232"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.633684 1142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:56.641984 1142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:56.644299 1142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:56.644469 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:56.644690 1142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:56.645027 1142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:56.645083 1142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.645128 1142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:56.645159 1142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.651655 1142 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:45393
I20260929 18:52:56.651690 1270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:45393 every 8 connection(s)
I20260929 18:52:56.652047 1142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:52:56 dist-test-slave-q2kh krb5kdc[1035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:56.660053 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1142
I20260929 18:52:56.660168 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:56.660364 29982 external_mini_cluster.cc:1428] Setting key 43932a024f3fb617cc00159bf0fd2e56
I20260929 18:52:56.661813 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:56.652757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36577 (local address 127.29.71.129:59901)
0929 18:52:56.652930 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:56.652972 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:56.653871 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:56.654276 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:56.654280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:56.654532 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:56.654716 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:56.654720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.655420 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.655421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:56.656224 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:56.656226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.656615 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.656615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:56.656711 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:56.657227 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:56.657236 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:56.658679 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:56.660292 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:56.660305 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:56.660307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:56.660622 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:56.660836 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:56.660837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:56.660837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:56.660983 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:56.661309 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:56.661314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:56.661438 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 18:52:56.661526 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:52:56.661649 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260929 18:52:56.661821 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:56.652802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:59901 (local address 127.29.71.190:36577)
0929 18:52:56.652937 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:56.652939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:56.653046 (+ 107us) server_negotiation.cc:407] Connection header received
0929 18:52:56.654024 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:56.654026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:56.654041 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:56.654126 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:56.654807 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.655338 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.656322 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.656535 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.656600 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:56.658778 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:56.658779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:56.658779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:56.658785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:56.660150 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:56.660694 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:56.660695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:56.660695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:56.660770 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:56.661077 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:56.661078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:56.661078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:56.661241 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:56.661315 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:56.661573 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:52:56.661671 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260929 18:52:56.662691 1271 heartbeater.cc:347] Connected to a master server at 127.29.71.190:36577
I20260929 18:52:56.662804 1271 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:56.663013 1271 heartbeater.cc:511] Master 127.29.71.190:36577 requested a full tablet report, sending...
I20260929 18:52:56.663522 1080 ts_manager.cc:194] Registered new tserver with Master: 2fbf8f760b454d039dddb2ea688eeb8c (127.29.71.129:45393)
I20260929 18:52:56.664127 1080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:59901
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:56.685797 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36577
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:56.764573 1277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:56.764770 1277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:56.764792 1277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:56.765941 1277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:56.765982 1277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:56.766008 1277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:56.767506 1277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:56.767779 1277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:56.768016 1277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:56.772974 1285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:56.773051 1283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:56.772974 1282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:56.773195 1277 server_base.cc:1034] running on GCE node
I20260929 18:52:56.773402 1277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:56.773594 1277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:56.774726 1277 hybrid_clock.cc:648] HybridClock initialized: now 1790707976774712 us; error 27 us; skew 500 ppm
Sep 29 18:52:56 dist-test-slave-q2kh krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:56.777189 1277 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:56.777657 1277 webserver.cc:460] Webserver started at http://127.29.71.130:41689/ using document root <none> and password file <none>
I20260929 18:52:56.777894 1277 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:56.777968 1277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:56.778115 1277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:56.779162 1277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "19f2335e0e474fb4a8b4d002139ac7af"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "74e36563b51e035db31bcec4d683a919"
server_key_iv: "17e4f40c56c05d6a81f02ca895562bd6"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.779512 1277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "19f2335e0e474fb4a8b4d002139ac7af"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "74e36563b51e035db31bcec4d683a919"
server_key_iv: "17e4f40c56c05d6a81f02ca895562bd6"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.780802 1277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:56.781574 1292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.781754 1277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:56.781852 1277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "19f2335e0e474fb4a8b4d002139ac7af"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "74e36563b51e035db31bcec4d683a919"
server_key_iv: "17e4f40c56c05d6a81f02ca895562bd6"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.781937 1277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:56.803614 1277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:56.805553 1277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:56.805724 1277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:56.805990 1277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:56.806380 1277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:56.806455 1277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.806501 1277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:56.806527 1277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.812335 1277 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:42327
I20260929 18:52:56.812372 1405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:42327 every 8 connection(s)
I20260929 18:52:56.812743 1277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:52:56 dist-test-slave-q2kh krb5kdc[1035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:56.820542 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1277
I20260929 18:52:56.820647 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:56.820858 29982 external_mini_cluster.cc:1428] Setting key 5ec94f499f3429779931e4eefca98333
I20260929 18:52:56.821944 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:56.813507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:39537 (local address 127.29.71.190:36577)
0929 18:52:56.813578 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:56.813580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:56.813749 (+ 169us) server_negotiation.cc:407] Connection header received
0929 18:52:56.814672 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:56.814674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:56.814683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:56.814752 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:56.815377 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.815872 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.816856 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.817047 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.817097 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:56.819234 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:56.819235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:56.819235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:56.819241 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:56.820422 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:56.820927 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:56.820929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:56.820929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:56.820992 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:56.821274 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:56.821275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:56.821275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:56.821401 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:56.821467 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:56.821794 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 18:52:56.821863 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:52:56.822263 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:56.813470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36577 (local address 127.29.71.130:39537)
0929 18:52:56.813627 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:56.813668 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:56.814586 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:56.814824 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:56.814828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:56.815102 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:56.815301 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:56.815305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.815969 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.815970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:56.816784 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:56.816786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.817117 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.817117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:56.817211 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:56.817741 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:56.817748 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:56.819153 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:56.820504 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:56.820511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:56.820514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:56.820818 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:56.821051 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:56.821052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:56.821052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:56.821188 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:56.821465 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:56.821469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:56.821654 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 18:52:56.821770 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:52:56.821959 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260929 18:52:56.822716 1406 heartbeater.cc:347] Connected to a master server at 127.29.71.190:36577
I20260929 18:52:56.822818 1406 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:56.823004 1406 heartbeater.cc:511] Master 127.29.71.190:36577 requested a full tablet report, sending...
I20260929 18:52:56.823376 1080 ts_manager.cc:194] Registered new tserver with Master: 19f2335e0e474fb4a8b4d002139ac7af (127.29.71.130:42327)
I20260929 18:52:56.823900 1080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:39537
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:56.846038 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36577
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:56.928053 1411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:56.928308 1411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:56.928344 1411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:56.929549 1411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:56.929620 1411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:56.929663 1411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:56.931195 1411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:56.931468 1411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:56.931702 1411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:56.937194 1419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:56.937206 1417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:56.937197 1416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:56.937350 1411 server_base.cc:1034] running on GCE node
I20260929 18:52:56.937589 1411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:56.937839 1411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:56.938998 1411 hybrid_clock.cc:648] HybridClock initialized: now 1790707976938984 us; error 26 us; skew 500 ppm
Sep 29 18:52:56 dist-test-slave-q2kh krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:56.941339 1411 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:56.941758 1411 webserver.cc:460] Webserver started at http://127.29.71.131:46675/ using document root <none> and password file <none>
I20260929 18:52:56.942015 1411 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:56.942086 1411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:56.942251 1411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:56.943280 1411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "63a4bb2e4582402b8316d96312c48e0c"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "a5f566bde55be38cd0afaad78afa66ee"
server_key_iv: "39dca3c6137474ea4fe00a927d0d607d"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.943641 1411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "63a4bb2e4582402b8316d96312c48e0c"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "a5f566bde55be38cd0afaad78afa66ee"
server_key_iv: "39dca3c6137474ea4fe00a927d0d607d"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.944913 1411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:56.945629 1426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.945797 1411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:56.945906 1411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "63a4bb2e4582402b8316d96312c48e0c"
format_stamp: "Formatted at 2026-09-29 18:52:56 on dist-test-slave-q2kh"
server_key: "a5f566bde55be38cd0afaad78afa66ee"
server_key_iv: "39dca3c6137474ea4fe00a927d0d607d"
server_key_version: "encryptionkey@0"
I20260929 18:52:56.945979 1411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:56.954375 1411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:56.956162 1411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:56.956319 1411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:56.956561 1411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:56.956864 1411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:56.956918 1411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.956962 1411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:56.956986 1411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:56.962889 1411 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:34967
I20260929 18:52:56.962956 1539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:34967 every 8 connection(s)
I20260929 18:52:56.963289 1411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:52:56 dist-test-slave-q2kh krb5kdc[1035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:56.971428 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1411
I20260929 18:52:56.971518 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:56.971695 29982 external_mini_cluster.cc:1428] Setting key 8fdf4c97cf71c9a6fa8580fda0d04cc4
I20260929 18:52:56.972937 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:56.964060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:37713 (local address 127.29.71.190:36577)
0929 18:52:56.964122 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:56.964124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:56.964310 (+ 186us) server_negotiation.cc:407] Connection header received
0929 18:52:56.965227 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:56.965228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:56.965236 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:56.965296 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:56.965844 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.966401 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.967469 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.967658 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.967716 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:56.970149 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:56.970195 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:56.970195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:56.970205 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:56.971508 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:56.971972 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:56.971974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:56.971974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:56.972035 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:56.972290 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:56.972291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:56.972291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:56.972414 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:56.972494 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:56.972766 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 18:52:56.972846 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:52:56.973138 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:56.964030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:36577 (local address 127.29.71.131:37713)
0929 18:52:56.964175 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:56.964216 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:56.965122 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:56.965349 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:56.965352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:56.965593 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:56.965776 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:56.965780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.966488 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.966488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:56.967398 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:56.967400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:56.967730 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:56.967731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:56.967879 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:56.968594 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:56.968601 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:56.970054 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:56.971582 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:56.971590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:56.971592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:56.971866 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:56.972104 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:56.972105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:56.972106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:56.972220 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:56.972484 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:56.972490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:56.972635 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 18:52:56.972744 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:52:56.972910 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260929 18:52:56.973745 1540 heartbeater.cc:347] Connected to a master server at 127.29.71.190:36577
I20260929 18:52:56.973838 1540 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:56.973999 1540 heartbeater.cc:511] Master 127.29.71.190:36577 requested a full tablet report, sending...
I20260929 18:52:56.974375 1080 ts_manager.cc:194] Registered new tserver with Master: 63a4bb2e4582402b8316d96312c48e0c (127.29.71.131:34967)
I20260929 18:52:56.974761 1080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:37713
I20260929 18:52:56.983781 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:52:56 dist-test-slave-q2kh krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:52:57 dist-test-slave-q2kh krb5kdc[1035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:52:57.007632 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:56.999725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.29.71.129:45393)
0929 18:52:56.999894 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:56.999904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:56.999915 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:52:56.999966 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:56.999968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:56.999986 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:57.000056 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:57.000626 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:57.001353 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:57.002077 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:57.002473 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:57.002531 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:57.004857 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:57.004867 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:57.004867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:57.004874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:57.006631 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:57.006957 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:57.006959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:57.006960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:57.007012 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:57.007174 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:57.007174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:57.007175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:57.007302 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:57.007431 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:57.007485 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 18:52:57.007550 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
Sep 29 18:52:57 dist-test-slave-q2kh krb5kdc[1035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:57.015837 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:57.008992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.29.71.190:36577)
0929 18:52:57.009050 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:57.009051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:57.009080 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:52:57.009147 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:57.009148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:57.009157 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:57.009205 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:57.009673 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:57.010241 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:57.011032 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:57.011216 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:57.011273 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:57.013505 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:57.013507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:57.013508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:57.013514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:57.014855 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:57.015194 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:57.015195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:57.015196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:57.015246 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:57.015410 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:57.015411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:57.015411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:57.015553 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:57.015631 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:57.015711 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 18:52:57.015771 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 18:52:57.016187 1080 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478
I20260929 18:52:57.016554 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1142
I20260929 18:52:57.021310 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1277
I20260929 18:52:57.025564 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1411
I20260929 18:52:57.029546 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1048
2026-09-29T18:52:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5203 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:52:57 dist-test-slave-q2kh krb5kdc[1552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:52:57 dist-test-slave-q2kh krb5kdc[1552](info): set up 2 sockets
Sep 29 18:52:57 dist-test-slave-q2kh krb5kdc[1552](info): commencing operation
krb5kdc: starting...
W20260929 18:52:58.066242 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:52:58.185877 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35933
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:35933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:58.266783 1565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:58.267047 1565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:58.267089 1565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:58.268249 1565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:58.268314 1565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:58.268339 1565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:58.268366 1565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:58.268396 1565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:58.269860 1565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:35933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:58.270143 1565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:58.270391 1565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:58.275120 1573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.275139 1571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.275182 1570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:58.275485 1565 server_base.cc:1034] running on GCE node
I20260929 18:52:58.275723 1565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:58.276005 1565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:58.277146 1565 hybrid_clock.cc:648] HybridClock initialized: now 1790707978277133 us; error 29 us; skew 500 ppm
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:58.279608 1565 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:52:58.280000 1565 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:58.280076 1565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:58.280232 1565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:58.281306 1565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "85f823f55f49428e93dcc6af1bba7847"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "a1bc6f6512a2885b9b65f5a744e57a92"
server_key_iv: "3f727494b9233a825a27bebc9c975a43"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.281672 1565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "85f823f55f49428e93dcc6af1bba7847"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "a1bc6f6512a2885b9b65f5a744e57a92"
server_key_iv: "3f727494b9233a825a27bebc9c975a43"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.283093 1565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:58.283814 1579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.284000 1565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.284103 1565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "85f823f55f49428e93dcc6af1bba7847"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "a1bc6f6512a2885b9b65f5a744e57a92"
server_key_iv: "3f727494b9233a825a27bebc9c975a43"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.284176 1565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:58.306114 1565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:58.308084 1565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:58.308243 1565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:58.312533 1565 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:35933
I20260929 18:52:58.312561 1631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:35933 every 8 connection(s)
I20260929 18:52:58.312850 1565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:58.313760 1632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:58.316078 1632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847: Bootstrap starting.
I20260929 18:52:58.316705 1632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:58.317011 1632 log.cc:824] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:58.317759 1632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847: No bootstrap required, opened a new log
I20260929 18:52:58.319586 1632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f823f55f49428e93dcc6af1bba7847" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35933 } }
I20260929 18:52:58.319732 1632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:58.319757 1632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f823f55f49428e93dcc6af1bba7847, State: Initialized, Role: FOLLOWER
I20260929 18:52:58.319950 1632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f823f55f49428e93dcc6af1bba7847" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35933 } }
I20260929 18:52:58.320036 1632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:58.320088 1632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:58.320142 1632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:58.320947 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1565
I20260929 18:52:58.321072 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:58.321285 29982 external_mini_cluster.cc:1428] Setting key 8b96454f3888a271b14fdf8d6ecf50b8
I20260929 18:52:58.321278 1632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f823f55f49428e93dcc6af1bba7847" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35933 } }
I20260929 18:52:58.321386 1632 leader_election.cc:302] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f823f55f49428e93dcc6af1bba7847; no voters:
I20260929 18:52:58.321595 1632 leader_election.cc:288] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:58.321630 1635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:58.321755 1635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [term 1 LEADER]: Becoming Leader. State: Replica: 85f823f55f49428e93dcc6af1bba7847, State: Running, Role: LEADER
I20260929 18:52:58.321893 1632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:58.321925 1635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f823f55f49428e93dcc6af1bba7847" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35933 } }
I20260929 18:52:58.322402 1636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85f823f55f49428e93dcc6af1bba7847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f823f55f49428e93dcc6af1bba7847" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35933 } } }
I20260929 18:52:58.322420 1637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f823f55f49428e93dcc6af1bba7847. Latest consensus state: current_term: 1 leader_uuid: "85f823f55f49428e93dcc6af1bba7847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f823f55f49428e93dcc6af1bba7847" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35933 } } }
I20260929 18:52:58.322616 1636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:58.322630 1637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:58.323045 1646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:58.323599 1646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:58.325333 1646 catalog_manager.cc:1347] Generated new cluster ID: f94e9fe2320844e585fac4f4f7ead5ad
I20260929 18:52:58.325387 1646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:58.334677 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.321960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.29.71.190:35933)
0929 18:52:58.322317 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.322328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.322366 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:52:58.323036 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.323044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.323125 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.323327 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:58.324229 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.325834 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.327024 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.327715 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.327810 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.330776 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:58.330780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:58.330781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:58.330794 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:58.332876 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.333294 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.333310 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.333310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.333416 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.333671 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.333671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.333672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.333991 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:58.334140 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.334273 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.334421 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":121,"threads_started":1}
I20260929 18:52:58.351786 1646 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:58.352456 1646 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:58.359054 1646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847: Generated new TSK 0
I20260929 18:52:58.359262 1646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:52:58.388835 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35933
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:58.467878 1659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:58.468075 1659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:58.468097 1659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:58.469165 1659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:58.469197 1659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:58.469223 1659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:58.470718 1659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:58.470906 1659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:58.471097 1659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:58.476140 1665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.476125 1667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.476147 1664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:58.476127 1659 server_base.cc:1034] running on GCE node
I20260929 18:52:58.476634 1659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:58.476878 1659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:58.478041 1659 hybrid_clock.cc:648] HybridClock initialized: now 1790707978478033 us; error 38 us; skew 500 ppm
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:58.480583 1659 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:52:58.480932 1659 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:58.481014 1659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:58.481161 1659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:58.482267 1659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "99df00ab91714ba7babc9c86e83007f5"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "975b7541a332ec11d6329cf6e2cbff89"
server_key_iv: "c4d574a4963d96fb864dd5ed39d05b47"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.482615 1659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "99df00ab91714ba7babc9c86e83007f5"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "975b7541a332ec11d6329cf6e2cbff89"
server_key_iv: "c4d574a4963d96fb864dd5ed39d05b47"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.483865 1659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:52:58.484551 1673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.484712 1659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:58.484814 1659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "99df00ab91714ba7babc9c86e83007f5"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "975b7541a332ec11d6329cf6e2cbff89"
server_key_iv: "c4d574a4963d96fb864dd5ed39d05b47"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.484885 1659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:58.494088 1659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:58.496428 1659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:58.496608 1659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:58.496830 1659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:58.497151 1659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:58.497206 1659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.497247 1659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:58.497275 1659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.503551 1659 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:40951
I20260929 18:52:58.503592 1786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:40951 every 8 connection(s)
I20260929 18:52:58.503859 1659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:58.512928 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.504590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:46193 (local address 127.29.71.190:35933)
0929 18:52:58.504651 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.504654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.504871 (+ 217us) server_negotiation.cc:407] Connection header received
0929 18:52:58.505365 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.505366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.505375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.505441 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:58.506449 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.506950 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.507893 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.508101 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.508151 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.510282 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:58.510289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:58.510289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:58.510296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:58.511549 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.511978 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.511979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.511980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.512037 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.512282 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.512283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.512284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.512396 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:58.512477 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.512741 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.512827 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:52:58.513106 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.504577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35933 (local address 127.29.71.129:46193)
0929 18:52:58.504739 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.504781 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.505269 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.505539 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.505543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.505792 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:58.506363 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.506368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.507026 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.507027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.507824 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.507826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.508172 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.508172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.508270 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.508795 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:58.508803 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:58.510116 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:58.511646 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:58.511652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:58.511654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:58.511870 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:58.512104 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:58.512105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:58.512106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:58.512215 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:58.512466 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:58.512472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:58.512611 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.512720 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.512872 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260929 18:52:58.513399 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1659
I20260929 18:52:58.513514 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:58.513669 1787 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35933
I20260929 18:52:58.513713 29982 external_mini_cluster.cc:1428] Setting key bd715f6b8918c63bfc18b6dcc8e1d5a3
I20260929 18:52:58.513780 1787 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:58.513926 1787 heartbeater.cc:511] Master 127.29.71.190:35933 requested a full tablet report, sending...
I20260929 18:52:58.514606 1596 ts_manager.cc:194] Registered new tserver with Master: 99df00ab91714ba7babc9c86e83007f5 (127.29.71.129:40951)
I20260929 18:52:58.515226 1596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:46193
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:52:58.539016 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35933
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:58.620611 1792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:58.620841 1792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:58.620880 1792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:58.622193 1792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:58.622256 1792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:58.622301 1792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:58.623903 1792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:58.624173 1792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:58.624400 1792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:58.629273 1797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.629278 1798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.629278 1800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:58.629539 1792 server_base.cc:1034] running on GCE node
I20260929 18:52:58.629766 1792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:58.630031 1792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:58.631182 1792 hybrid_clock.cc:648] HybridClock initialized: now 1790707978631167 us; error 28 us; skew 500 ppm
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:58.633848 1792 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:52:58.634322 1792 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:58.634397 1792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:58.634544 1792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:58.635613 1792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "d2e917e79ee24c1cad23c09536d329cd"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "b146774236a9e75cb7db4e097bae602f"
server_key_iv: "a9e5d54ed4b02990cdd561f4542e069e"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.635993 1792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "d2e917e79ee24c1cad23c09536d329cd"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "b146774236a9e75cb7db4e097bae602f"
server_key_iv: "a9e5d54ed4b02990cdd561f4542e069e"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.637368 1792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:52:58.638352 1806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.638553 1792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:58.638657 1792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "d2e917e79ee24c1cad23c09536d329cd"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "b146774236a9e75cb7db4e097bae602f"
server_key_iv: "a9e5d54ed4b02990cdd561f4542e069e"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.638751 1792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:58.647500 1792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:58.649775 1792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:58.649943 1792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:58.650224 1792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:58.650534 1792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:58.650588 1792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.650631 1792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:58.650655 1792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.656399 1792 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:46775
I20260929 18:52:58.656438 1919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:46775 every 8 connection(s)
I20260929 18:52:58.656714 1792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:58.663797 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1792
I20260929 18:52:58.663908 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:58.664125 29982 external_mini_cluster.cc:1428] Setting key 9b6c5d681c83cd769df1642351844a05
I20260929 18:52:58.666234 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.657457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:42045 (local address 127.29.71.190:35933)
0929 18:52:58.657543 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.657546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.657660 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:52:58.658286 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.658287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.658308 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.658362 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:58.659289 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.659819 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.660855 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.661049 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.661134 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.663241 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:58.663242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:58.663242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:58.663250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:58.664711 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.665135 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.665136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.665137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.665194 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.665419 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.665420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.665420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.665564 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:58.665636 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.665974 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.666068 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:52:58.666354 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.657382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35933 (local address 127.29.71.130:42045)
0929 18:52:58.657533 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.657570 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.658132 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.658435 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.658438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.658703 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:58.659180 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.659185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.659911 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.659912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.660729 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.660731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.661143 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.661144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.661264 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.661719 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:58.661727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:58.663132 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:58.664803 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:58.664810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:58.664812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:58.665062 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:58.665260 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:58.665261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:58.665261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:58.665352 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:58.665656 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:58.665662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:58.665827 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.665935 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.666074 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260929 18:52:58.666903 1920 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35933
I20260929 18:52:58.667037 1920 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:58.667182 1920 heartbeater.cc:511] Master 127.29.71.190:35933 requested a full tablet report, sending...
I20260929 18:52:58.667616 1596 ts_manager.cc:194] Registered new tserver with Master: d2e917e79ee24c1cad23c09536d329cd (127.29.71.130:46775)
I20260929 18:52:58.668144 1596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:42045
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:52:58.689868 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35933
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:52:58.769555 1925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:58.769753 1925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:58.769776 1925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:58.770861 1925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:58.770897 1925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:58.770923 1925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:58.772325 1925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:58.772517 1925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:58.772679 1925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:58.777457 1930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.777467 1933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:58.777467 1931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:58.777709 1925 server_base.cc:1034] running on GCE node
I20260929 18:52:58.777944 1925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:58.778258 1925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:58.779431 1925 hybrid_clock.cc:648] HybridClock initialized: now 1790707978779424 us; error 44 us; skew 500 ppm
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:52:58.781911 1925 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:52:58.782318 1925 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:58.782392 1925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:58.782546 1925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:58.783591 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "b145ab005c70449b88ca4700b56da785"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "d11652c355dfd74d0d26d4f268dde717"
server_key_iv: "98d9493353b816f0acb6b9589e709345"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.783965 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "b145ab005c70449b88ca4700b56da785"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "d11652c355dfd74d0d26d4f268dde717"
server_key_iv: "98d9493353b816f0acb6b9589e709345"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.785434 1925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:52:58.786310 1939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.786521 1925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:52:58.786612 1925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "b145ab005c70449b88ca4700b56da785"
format_stamp: "Formatted at 2026-09-29 18:52:58 on dist-test-slave-q2kh"
server_key: "d11652c355dfd74d0d26d4f268dde717"
server_key_iv: "98d9493353b816f0acb6b9589e709345"
server_key_version: "encryptionkey@0"
I20260929 18:52:58.786701 1925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:58.804354 1925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:58.806732 1925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:58.806900 1925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:58.807137 1925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:58.807451 1925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:58.807505 1925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.807548 1925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:58.807577 1925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:58.813158 1925 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:34693
I20260929 18:52:58.813198 2052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:34693 every 8 connection(s)
I20260929 18:52:58.813457 1925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:52:58.814567 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 1925
I20260929 18:52:58.814679 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:58.814857 29982 external_mini_cluster.cc:1428] Setting key fb3c78e97ff5fd67270cfed842f7cd3d
Sep 29 18:52:58 dist-test-slave-q2kh krb5kdc[1552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:52:58.822495 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.814127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:58919 (local address 127.29.71.190:35933)
0929 18:52:58.814194 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.814196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.814409 (+ 213us) server_negotiation.cc:407] Connection header received
0929 18:52:58.814985 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.814987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.815001 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.815062 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:58.816220 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.816736 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.817691 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.817891 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.817950 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.820084 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:58.820086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:58.820086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:58.820093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:58.821232 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.821599 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.821600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.821600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.821646 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.821854 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.821855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.821855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.821968 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:58.822018 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.822334 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.822414 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:52:58.822721 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.814108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35933 (local address 127.29.71.131:58919)
0929 18:52:58.814308 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.814343 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.814877 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.815173 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.815176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.815530 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:58.816078 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.816083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.816818 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.816819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.817623 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.817625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.817969 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.817970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.818066 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.818622 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:52:58.818635 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:58.820000 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:58.821306 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:58.821312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:58.821313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:58.821535 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:58.821703 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:52:58.821704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:52:58.821704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:52:58.821801 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:52:58.822024 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:58.822028 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:58.822229 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.822316 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.822429 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":122,"threads_started":1}
I20260929 18:52:58.823431 2053 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35933
I20260929 18:52:58.823525 2053 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:58.823674 2053 heartbeater.cc:511] Master 127.29.71.190:35933 requested a full tablet report, sending...
I20260929 18:52:58.824030 1596 ts_manager.cc:194] Registered new tserver with Master: b145ab005c70449b88ca4700b56da785 (127.29.71.131:34693)
I20260929 18:52:58.824402 1596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:58919
I20260929 18:52:58.826565 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:52:58.833294 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.827854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.29.71.190:35933)
0929 18:52:58.827906 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.827907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.828000 (+ 93us) server_negotiation.cc:407] Connection header received
0929 18:52:58.828052 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.828052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.828060 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.828106 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:58.828628 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.829081 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.829760 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.829940 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.829988 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.831076 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:58.831077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:58.831077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:52:58.831082 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:58.832147 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.832447 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.832448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.832448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.832495 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:52:58.832657 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:52:58.832658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:52:58.832658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:52:58.832782 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:58.832860 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.833179 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.833235 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:52:58.835039 1596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:52:58.836699 1596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:58.843264 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.838962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46775 (local address 127.0.0.1:58262)
0929 18:52:58.839236 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.839244 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.839280 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.839549 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.839552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.839555 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:58.839625 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.839626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.841116 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.841117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.842060 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.842062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.842911 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.842912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.843002 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.843017 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.843103 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.843160 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260929 18:52:58.843262 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.839175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.29.71.130:46775)
0929 18:52:58.839357 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.839362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.839372 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:58.839415 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.839417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.839435 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.839520 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:58.839732 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.840979 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.842122 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.842816 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.842893 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.842945 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.843125 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.843164 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260929 18:52:58.843540 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.838872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:34693 (local address 127.0.0.1:36308)
0929 18:52:58.839049 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.839080 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.839143 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.839590 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.839591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.839593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:58.839628 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.839667 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.840862 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.840862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.842403 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.842405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.843250 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.843250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.843331 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.843336 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.843374 (+ 38us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.843427 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":41,"thread_start_us":66,"threads_started":1}
I20260929 18:52:58.843609 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.838873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:40951 (local address 127.0.0.1:53010)
0929 18:52:58.839041 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.839080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.839143 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.839580 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.839582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.839584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:58.839636 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.839638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.841045 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.841046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.842369 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.842371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.843297 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.843298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.843367 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.843371 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.843404 (+ 33us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.843443 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":92,"thread_start_us":71,"threads_started":1}
I20260929 18:52:58.843768 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.839237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.29.71.129:40951)
0929 18:52:58.839405 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.839408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.839417 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:58.839452 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.839454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.839470 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.839733 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:58.839822 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.840944 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.842491 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.843195 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.843537 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.843601 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.843627 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.843668 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260929 18:52:58.843918 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.839053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.29.71.131:34693)
0929 18:52:58.839373 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.839378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.839390 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:52:58.839429 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.839431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.839449 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.839750 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:58.839876 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.840733 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.842501 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.843153 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.843698 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.843766 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.843794 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.843837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":80,"threads_started":1}
I20260929 18:52:58.844349 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 757c1abb60434f8db9e3d9867a05467b (DEFAULT_TABLE table=test-table [id=e778a1f84802429faf8ef80e0ce5be71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:58.844350 1854 tablet_service.cc:1467] Processing CreateTablet for tablet 757c1abb60434f8db9e3d9867a05467b (DEFAULT_TABLE table=test-table [id=e778a1f84802429faf8ef80e0ce5be71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:58.844564 1987 tablet_service.cc:1467] Processing CreateTablet for tablet 757c1abb60434f8db9e3d9867a05467b (DEFAULT_TABLE table=test-table [id=e778a1f84802429faf8ef80e0ce5be71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:58.844703 1854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757c1abb60434f8db9e3d9867a05467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:58.844703 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757c1abb60434f8db9e3d9867a05467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:58.844918 1987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757c1abb60434f8db9e3d9867a05467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:58.847643 2067 tablet_bootstrap.cc:492] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Bootstrap starting.
I20260929 18:52:58.848059 2068 tablet_bootstrap.cc:492] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: Bootstrap starting.
I20260929 18:52:58.848382 2067 tablet_bootstrap.cc:654] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:58.848747 2068 tablet_bootstrap.cc:654] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:58.849108 2067 log.cc:824] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:58.849339 2068 log.cc:824] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:58.849659 2069 tablet_bootstrap.cc:492] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Bootstrap starting.
I20260929 18:52:58.849984 2067 tablet_bootstrap.cc:492] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: No bootstrap required, opened a new log
I20260929 18:52:58.850044 2067 ts_tablet_manager.cc:1397] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:52:58.850296 2069 tablet_bootstrap.cc:654] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:58.850705 2068 tablet_bootstrap.cc:492] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: No bootstrap required, opened a new log
I20260929 18:52:58.850850 2069 log.cc:824] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:58.850893 2068 ts_tablet_manager.cc:1397] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:52:58.851588 2069 tablet_bootstrap.cc:492] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: No bootstrap required, opened a new log
I20260929 18:52:58.851676 2069 ts_tablet_manager.cc:1397] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:52:58.852470 2067 raft_consensus.cc:348] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.852665 2067 raft_consensus.cc:374] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:58.852702 2067 raft_consensus.cc:729] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b145ab005c70449b88ca4700b56da785, State: Initialized, Role: FOLLOWER
I20260929 18:52:58.852875 2067 consensus_queue.cc:260] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.853220 2067 ts_tablet_manager.cc:1428] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:58.853220 2053 heartbeater.cc:503] Master 127.29.71.190:35933 was elected leader, sending a full tablet report...
I20260929 18:52:58.853386 2068 raft_consensus.cc:348] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.853536 2068 raft_consensus.cc:374] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:58.853559 2068 raft_consensus.cc:729] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99df00ab91714ba7babc9c86e83007f5, State: Initialized, Role: FOLLOWER
I20260929 18:52:58.853585 2069 raft_consensus.cc:348] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.853684 2068 consensus_queue.cc:260] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.853763 2069 raft_consensus.cc:374] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:58.853803 2069 raft_consensus.cc:729] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e917e79ee24c1cad23c09536d329cd, State: Initialized, Role: FOLLOWER
I20260929 18:52:58.853899 2069 consensus_queue.cc:260] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.853994 1787 heartbeater.cc:503] Master 127.29.71.190:35933 was elected leader, sending a full tablet report...
I20260929 18:52:58.854007 2068 ts_tablet_manager.cc:1428] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:58.854203 2069 ts_tablet_manager.cc:1428] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 18:52:58.854375 1920 heartbeater.cc:503] Master 127.29.71.190:35933 was elected leader, sending a full tablet report...
W20260929 18:52:58.907249 1921 tablet.cc:2369] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:52:58.983449 2073 raft_consensus.cc:482] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:58.983570 2073 raft_consensus.cc:504] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.983975 2073 leader_election.cc:288] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99df00ab91714ba7babc9c86e83007f5 (127.29.71.129:40951), d2e917e79ee24c1cad23c09536d329cd (127.29.71.130:46775)
I20260929 18:52:58.988016 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.984261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:40951 (local address 127.29.71.131:55487)
0929 18:52:58.984325 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.984338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.984387 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.984556 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.984557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.984559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:58.984633 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.984634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.985606 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.985608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.987007 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.987009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.987727 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.987728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.987821 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.987824 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.987885 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.987917 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":138}
I20260929 18:52:58.988055 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.984327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:55487 (local address 127.29.71.129:40951)
0929 18:52:58.984374 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.984376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.984390 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:52:58.984453 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.984454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.984461 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.984543 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:58.984716 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.985490 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.987092 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.987653 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.987711 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.987717 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.987903 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.987965 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:52:58.988129 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.984262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46775 (local address 127.29.71.131:54831)
0929 18:52:58.984480 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:58.984489 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:58.984540 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:58.984705 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:58.984706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:58.984708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:52:58.984766 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.984767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.985606 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.985608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.987007 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:58.987009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.987814 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.987815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:58.987910 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.987915 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:52:58.987969 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:52:58.988007 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":40,"thread_start_us":132,"threads_started":1}
I20260929 18:52:58.988308 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:58.984335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:54831 (local address 127.29.71.130:46775)
0929 18:52:58.984394 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:58.984397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:58.984595 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:52:58.984622 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:58.984623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:58.984635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:58.984690 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:52:58.984835 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.985523 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.987143 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:58.987727 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:58.987816 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:58.987824 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:58.987995 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:52:58.988037 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:52:58.988449 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c1abb60434f8db9e3d9867a05467b" candidate_uuid: "b145ab005c70449b88ca4700b56da785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df00ab91714ba7babc9c86e83007f5" is_pre_election: true
I20260929 18:52:58.988533 1874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c1abb60434f8db9e3d9867a05467b" candidate_uuid: "b145ab005c70449b88ca4700b56da785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e917e79ee24c1cad23c09536d329cd" is_pre_election: true
I20260929 18:52:58.988631 1741 raft_consensus.cc:2454] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b145ab005c70449b88ca4700b56da785 in term 0.
I20260929 18:52:58.988667 1874 raft_consensus.cc:2454] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b145ab005c70449b88ca4700b56da785 in term 0.
I20260929 18:52:58.988853 1940 leader_election.cc:302] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99df00ab91714ba7babc9c86e83007f5, b145ab005c70449b88ca4700b56da785; no voters:
I20260929 18:52:58.988984 2073 raft_consensus.cc:2781] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:58.989039 2073 raft_consensus.cc:482] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:58.989063 2073 raft_consensus.cc:3037] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:58.989720 2073 raft_consensus.cc:504] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.989918 2073 leader_election.cc:288] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [CANDIDATE]: Term 1 election: Requested vote from peers 99df00ab91714ba7babc9c86e83007f5 (127.29.71.129:40951), d2e917e79ee24c1cad23c09536d329cd (127.29.71.130:46775)
I20260929 18:52:58.989998 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c1abb60434f8db9e3d9867a05467b" candidate_uuid: "b145ab005c70449b88ca4700b56da785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99df00ab91714ba7babc9c86e83007f5"
I20260929 18:52:58.990031 1874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757c1abb60434f8db9e3d9867a05467b" candidate_uuid: "b145ab005c70449b88ca4700b56da785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e917e79ee24c1cad23c09536d329cd"
I20260929 18:52:58.990077 1741 raft_consensus.cc:3037] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:58.990097 1874 raft_consensus.cc:3037] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:58.990834 1741 raft_consensus.cc:2454] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b145ab005c70449b88ca4700b56da785 in term 1.
I20260929 18:52:58.990945 1874 raft_consensus.cc:2454] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b145ab005c70449b88ca4700b56da785 in term 1.
I20260929 18:52:58.991041 1940 leader_election.cc:302] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99df00ab91714ba7babc9c86e83007f5, b145ab005c70449b88ca4700b56da785; no voters:
I20260929 18:52:58.991128 2073 raft_consensus.cc:2781] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:58.991269 2073 raft_consensus.cc:686] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 1 LEADER]: Becoming Leader. State: Replica: b145ab005c70449b88ca4700b56da785, State: Running, Role: LEADER
I20260929 18:52:58.991379 2073 consensus_queue.cc:237] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } }
I20260929 18:52:58.992211 1595 catalog_manager.cc:5495] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 reported cstate change: term changed from 0 to 1, leader changed from <none> to b145ab005c70449b88ca4700b56da785 (127.29.71.131). New cstate: current_term: 1 leader_uuid: "b145ab005c70449b88ca4700b56da785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b145ab005c70449b88ca4700b56da785" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 } health_report { overall_health: UNKNOWN } } }
W20260929 18:52:59.004580 1788 tablet.cc:2369] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:52:59.027938 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.025456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.29.71.131:34693)
0929 18:52:59.025520 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.025522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.025545 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:52:59.025585 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.025586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.025606 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.025690 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:52:59.025843 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.026343 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.027107 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.027321 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.027396 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:59.027576 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:52:59.027693 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:52:59.027792 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.027810 (+ 18us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.027831 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:52:59.029915 1741 raft_consensus.cc:1261] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5 [term 1 FOLLOWER]: Refusing update from remote peer b145ab005c70449b88ca4700b56da785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:59.029906 1874 raft_consensus.cc:1261] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 1 FOLLOWER]: Refusing update from remote peer b145ab005c70449b88ca4700b56da785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:59.030298 2077 consensus_queue.cc:1035] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99df00ab91714ba7babc9c86e83007f5" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:59.030486 2073 consensus_queue.cc:1035] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e917e79ee24c1cad23c09536d329cd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:59.033098 2081 mvcc.cc:205] Tried to move back new op lower bound from 7334739882103459840 to 7334739881949532160. Current Snapshot: MvccSnapshot[applied={T|T < 7334739882103459840}]
I20260929 18:52:59.033419 2082 mvcc.cc:205] Tried to move back new op lower bound from 7334739882103459840 to 7334739881949532160. Current Snapshot: MvccSnapshot[applied={T|T < 7334739882103459840}]
I20260929 18:52:59.035107 1595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:59.035198 1595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:59.036005 1595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f823f55f49428e93dcc6af1bba7847: Sending DeleteTablet for 3 replicas of tablet 757c1abb60434f8db9e3d9867a05467b
I20260929 18:52:59.036343 1987 tablet_service.cc:1514] Processing DeleteTablet for tablet 757c1abb60434f8db9e3d9867a05467b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:59 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:36308
I20260929 18:52:59.036365 1854 tablet_service.cc:1514] Processing DeleteTablet for tablet 757c1abb60434f8db9e3d9867a05467b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:59 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:58262
I20260929 18:52:59.036600 1719 tablet_service.cc:1514] Processing DeleteTablet for tablet 757c1abb60434f8db9e3d9867a05467b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:59 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:53010
I20260929 18:52:59.036835 2093 tablet_replica.cc:330] T 757c1abb60434f8db9e3d9867a05467b P 99df00ab91714ba7babc9c86e83007f5: stopping tablet replica
I20260929 18:52:59.036962 2091 tablet_replica.cc:330] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: stopping tablet replica
I20260929 18:52:59.037053 2091 raft_consensus.cc:2229] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:59.037151 2091 raft_consensus.cc:2258] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:59.037146 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1659
I20260929 18:52:59.038385 2091 ts_tablet_manager.cc:1905] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:59.040457 2091 ts_tablet_manager.cc:1918] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:52:59.040539 2091 log.cc:1196] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/757c1abb60434f8db9e3d9867a05467b
I20260929 18:52:59.040795 2091 ts_tablet_manager.cc:1939] T 757c1abb60434f8db9e3d9867a05467b P b145ab005c70449b88ca4700b56da785: Deleting consensus metadata
I20260929 18:52:59.040978 2092 tablet_replica.cc:330] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: stopping tablet replica
I20260929 18:52:59.041113 2092 raft_consensus.cc:2229] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:59.041486 2092 raft_consensus.cc:2258] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:59.041509 1583 catalog_manager.cc:4841] TS b145ab005c70449b88ca4700b56da785 (127.29.71.131:34693): tablet 757c1abb60434f8db9e3d9867a05467b (table test-table [id=e778a1f84802429faf8ef80e0ce5be71]) successfully deleted
I20260929 18:52:59.042016 2092 ts_tablet_manager.cc:1905] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:59.043876 2092 ts_tablet_manager.cc:1918] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:52:59.043953 2092 log.cc:1196] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/757c1abb60434f8db9e3d9867a05467b
I20260929 18:52:59.044219 2092 ts_tablet_manager.cc:1939] T 757c1abb60434f8db9e3d9867a05467b P d2e917e79ee24c1cad23c09536d329cd: Deleting consensus metadata
W20260929 18:52:59.044561 1580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:40951 (error 108)
I20260929 18:52:59.044643 1582 catalog_manager.cc:4841] TS d2e917e79ee24c1cad23c09536d329cd (127.29.71.130:46775): tablet 757c1abb60434f8db9e3d9867a05467b (table test-table [id=e778a1f84802429faf8ef80e0ce5be71]) successfully deleted
I20260929 18:52:59.045092 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.044956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:40951 (local address 127.0.0.1:53016)
0929 18:52:59.045005 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.045052 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 18:52:59.045114 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1792
W20260929 18:52:59.045238 1580 catalog_manager.cc:4584] TS 99df00ab91714ba7babc9c86e83007f5 (127.29.71.129:40951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 757c1abb60434f8db9e3d9867a05467b: Network error: Client connection negotiation failed: client connection to 127.29.71.129:40951: connect: Connection refused (error 111)
I20260929 18:52:59.051887 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1925
I20260929 18:52:59.056437 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 1565
2026-09-29T18:52:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2027 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T18:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:59Z Disabled control of system clock
I20260929 18:52:59.083396 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35985
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:35985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:52:59.168179 2098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:59.168407 2098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:59.168432 2098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:59.169603 2098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:59.169642 2098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:59.169656 2098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:59.169670 2098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:59.169683 2098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:52:59.171170 2098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:35985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:59.171391 2098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:59.171643 2098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:59.176508 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.176571 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:59.176779 2098 server_base.cc:1034] running on GCE node
W20260929 18:52:59.176566 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:59.177090 2098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:59.177388 2098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:59.178529 2098 hybrid_clock.cc:648] HybridClock initialized: now 1790707979178500 us; error 41 us; skew 500 ppm
I20260929 18:52:59.179811 2098 webserver.cc:460] Webserver started at http://127.29.71.190:38875/ using document root <none> and password file <none>
I20260929 18:52:59.180070 2098 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:59.180116 2098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:59.180276 2098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:59.181324 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "d35d32f313eb4ea0967df9fa5cd02334"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "541ebe975e972f39382bb55b48b9dfd9"
server_key_iv: "dc6255338a74e3cf2ca597de0ebf0113"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.181689 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "d35d32f313eb4ea0967df9fa5cd02334"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "541ebe975e972f39382bb55b48b9dfd9"
server_key_iv: "dc6255338a74e3cf2ca597de0ebf0113"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.183049 2098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:59.183734 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.183919 2098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:59.183996 2098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "d35d32f313eb4ea0967df9fa5cd02334"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "541ebe975e972f39382bb55b48b9dfd9"
server_key_iv: "dc6255338a74e3cf2ca597de0ebf0113"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.184089 2098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:59.187485 2098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:59.187745 2098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:59.187879 2098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:59.191570 2098 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:35985
I20260929 18:52:59.191615 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:35985 every 8 connection(s)
I20260929 18:52:59.191932 2098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:59.192555 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:59.195322 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334: Bootstrap starting.
I20260929 18:52:59.195899 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:59.196193 2165 log.cc:824] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:59.196873 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334: No bootstrap required, opened a new log
I20260929 18:52:59.197256 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2098
I20260929 18:52:59.197365 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:59.197551 29982 external_mini_cluster.cc:1428] Setting key 7e3494bd74bd051312019f716293f5f3
I20260929 18:52:59.199337 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35d32f313eb4ea0967df9fa5cd02334" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35985 } }
I20260929 18:52:59.199535 2165 raft_consensus.cc:374] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:59.199582 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35d32f313eb4ea0967df9fa5cd02334, State: Initialized, Role: FOLLOWER
I20260929 18:52:59.199741 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35d32f313eb4ea0967df9fa5cd02334" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35985 } }
I20260929 18:52:59.199831 2165 raft_consensus.cc:388] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:59.199914 2165 raft_consensus.cc:482] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:59.199976 2165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:59.200250 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.198176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.29.71.190:35985)
0929 18:52:59.198380 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.198386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.198410 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:52:59.198818 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.198825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.198861 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.198976 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.199723 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.199727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.199727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.199732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.199782 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.199797 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.199883 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.199993 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.200054 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260929 18:52:59.201064 2165 raft_consensus.cc:504] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35d32f313eb4ea0967df9fa5cd02334" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35985 } }
I20260929 18:52:59.201186 2165 leader_election.cc:302] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d35d32f313eb4ea0967df9fa5cd02334; no voters:
I20260929 18:52:59.201336 2165 leader_election.cc:288] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:59.201354 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:59.201563 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [term 1 LEADER]: Becoming Leader. State: Replica: d35d32f313eb4ea0967df9fa5cd02334, State: Running, Role: LEADER
I20260929 18:52:59.201640 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:59.201692 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35d32f313eb4ea0967df9fa5cd02334" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35985 } }
I20260929 18:52:59.202039 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35d32f313eb4ea0967df9fa5cd02334. Latest consensus state: current_term: 1 leader_uuid: "d35d32f313eb4ea0967df9fa5cd02334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35d32f313eb4ea0967df9fa5cd02334" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35985 } } }
I20260929 18:52:59.202035 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d35d32f313eb4ea0967df9fa5cd02334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35d32f313eb4ea0967df9fa5cd02334" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35985 } } }
I20260929 18:52:59.202184 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:59.202239 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334 [sys.catalog]: This master's current role is: LEADER
W20260929 18:52:59.203367 2186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:52:59.203424 2186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:52:59.203526 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:59.203711 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:59.205056 2179 catalog_manager.cc:1347] Generated new cluster ID: cf7a714dc3ce41e6b0d449eefbd4e1d1
I20260929 18:52:59.205130 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:52:59.220322 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:59.220526 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:59.227427 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334: Generated new TSK 0
I20260929 18:52:59.227628 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:52:59.237185 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35985
--builtin_ntp_servers=127.29.71.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:52:59.321447 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:59.321650 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:59.321672 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:59.322949 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:59.322990 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:59.323016 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:52:59.324335 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:59.324579 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:59.324892 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:59.330717 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.330739 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.330726 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:59.330925 2189 server_base.cc:1034] running on GCE node
I20260929 18:52:59.331178 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:59.331434 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:59.332621 2189 hybrid_clock.cc:648] HybridClock initialized: now 1790707979332610 us; error 26 us; skew 500 ppm
I20260929 18:52:59.333914 2189 webserver.cc:460] Webserver started at http://127.29.71.129:45189/ using document root <none> and password file <none>
I20260929 18:52:59.334265 2189 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:59.334316 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:59.334470 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:59.335537 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "ee9fb6bdc4b3cb6303a498f90b6c325a"
server_key_iv: "cf3da1687c13114ed4fe658b80aa360c"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.335894 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "ee9fb6bdc4b3cb6303a498f90b6c325a"
server_key_iv: "cf3da1687c13114ed4fe658b80aa360c"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.337193 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 18:52:59.337867 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.338068 2189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:59.338204 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "ee9fb6bdc4b3cb6303a498f90b6c325a"
server_key_iv: "cf3da1687c13114ed4fe658b80aa360c"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.338279 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:59.342304 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:59.342545 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:59.342638 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:59.342783 2189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:59.343024 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:59.343052 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.343072 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:59.343084 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.348444 2189 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:38561
I20260929 18:52:59.348517 2316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:38561 every 8 connection(s)
I20260929 18:52:59.348842 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:52:59.351948 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.349552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:44023 (local address 127.29.71.190:35985)
0929 18:52:59.349610 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.349611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.349848 (+ 237us) server_negotiation.cc:407] Connection header received
0929 18:52:59.350443 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.350444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.350456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.350506 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.351695 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.351696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.351696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.351701 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.351716 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.351720 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.351768 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.351865 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.351880 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:52:59.352155 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.349537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35985 (local address 127.29.71.129:44023)
0929 18:52:59.349735 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.349766 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.350323 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.350588 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.350592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.350608 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.351544 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.351545 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.351572 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.351601 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.351777 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.351781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.351784 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.351875 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.351941 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260929 18:52:59.352666 2317 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35985
I20260929 18:52:59.352767 2317 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:59.352846 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2189
I20260929 18:52:59.352939 2317 heartbeater.cc:511] Master 127.29.71.190:35985 requested a full tablet report, sending...
I20260929 18:52:59.352974 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:52:59.353164 29982 external_mini_cluster.cc:1428] Setting key c4b59c97ee99e149298eb2d321461870
I20260929 18:52:59.353510 2129 ts_manager.cc:194] Registered new tserver with Master: 182a7f50f6cf41c9ac25a9759c3ed0bb (127.29.71.129:38561)
I20260929 18:52:59.354286 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35985
--builtin_ntp_servers=127.29.71.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:52:59.435618 2320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:59.435828 2320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:59.435853 2320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:59.437072 2320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:59.437114 2320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:59.437141 2320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:52:59.438584 2320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:59.438844 2320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:59.439167 2320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:59.444416 2325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.444413 2328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.444422 2326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:59.444701 2320 server_base.cc:1034] running on GCE node
I20260929 18:52:59.444952 2320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:59.445222 2320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:59.446391 2320 hybrid_clock.cc:648] HybridClock initialized: now 1790707979446365 us; error 37 us; skew 500 ppm
I20260929 18:52:59.447690 2320 webserver.cc:460] Webserver started at http://127.29.71.130:35301/ using document root <none> and password file <none>
I20260929 18:52:59.447952 2320 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:59.448027 2320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:59.448179 2320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:59.449318 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "a126b43d717041679900e1bdd10fabe4"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "16847c51379a1a72cbdc213f9d98aafd"
server_key_iv: "f573d59c5a2e6153e374101610804006"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.449687 2320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "a126b43d717041679900e1bdd10fabe4"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "16847c51379a1a72cbdc213f9d98aafd"
server_key_iv: "f573d59c5a2e6153e374101610804006"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.451079 2320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:59.451797 2334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.451977 2320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:52:59.452080 2320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "a126b43d717041679900e1bdd10fabe4"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "16847c51379a1a72cbdc213f9d98aafd"
server_key_iv: "f573d59c5a2e6153e374101610804006"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.452152 2320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:59.455657 2320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:59.455920 2320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:59.456063 2320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:59.456267 2320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:59.456583 2320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:59.456636 2320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.456676 2320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:59.456701 2320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.462284 2320 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:42713
I20260929 18:52:59.462352 2447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:42713 every 8 connection(s)
I20260929 18:52:59.462684 2320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:52:59.465556 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.463451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:49543 (local address 127.29.71.190:35985)
0929 18:52:59.463495 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.463497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.463718 (+ 221us) server_negotiation.cc:407] Connection header received
0929 18:52:59.464181 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.464182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.464188 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.464253 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.465368 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.465369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.465369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.465373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.465387 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.465392 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.465440 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.465489 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.465501 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:52:59.465783 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.463411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35985 (local address 127.29.71.130:49543)
0929 18:52:59.463593 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.463635 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.464098 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.464310 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.464313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.464321 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.465221 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.465221 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.465250 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.465280 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.465440 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.465442 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.465445 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.465496 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.465544 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260929 18:52:59.466305 2448 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35985
I20260929 18:52:59.466380 2448 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:59.466506 2448 heartbeater.cc:511] Master 127.29.71.190:35985 requested a full tablet report, sending...
I20260929 18:52:59.466823 2128 ts_manager.cc:194] Registered new tserver with Master: a126b43d717041679900e1bdd10fabe4 (127.29.71.130:42713)
I20260929 18:52:59.468812 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2320
I20260929 18:52:59.468896 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:52:59.469038 29982 external_mini_cluster.cc:1428] Setting key 3cae567b1db03058e1f60b15b7b280d7
I20260929 18:52:59.469889 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35985
--builtin_ntp_servers=127.29.71.148:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:52:59.550045 2451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:59.550276 2451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:59.550302 2451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:59.551396 2451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:59.551429 2451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:59.551453 2451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:52:59.552680 2451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:59.552868 2451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:59.553139 2451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:52:59.558449 2457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.558457 2456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:59.558581 2451 server_base.cc:1034] running on GCE node
W20260929 18:52:59.558511 2459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:59.558836 2451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:59.559093 2451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:59.560235 2451 hybrid_clock.cc:648] HybridClock initialized: now 1790707979560217 us; error 41 us; skew 500 ppm
I20260929 18:52:59.561540 2451 webserver.cc:460] Webserver started at http://127.29.71.131:36401/ using document root <none> and password file <none>
I20260929 18:52:59.561779 2451 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:59.561827 2451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:59.562002 2451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:59.563125 2451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "28f8f11648c24672bd3c148621b73079"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "b0b9e2cb11884f69538dc5206eaf93f1"
server_key_iv: "d4dfce80dd46652b08ea7e29d349b6b8"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.563480 2451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "28f8f11648c24672bd3c148621b73079"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "b0b9e2cb11884f69538dc5206eaf93f1"
server_key_iv: "d4dfce80dd46652b08ea7e29d349b6b8"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.564822 2451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:52:59.565526 2465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.565701 2451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:59.565804 2451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "28f8f11648c24672bd3c148621b73079"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "b0b9e2cb11884f69538dc5206eaf93f1"
server_key_iv: "d4dfce80dd46652b08ea7e29d349b6b8"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.565876 2451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:59.569339 2451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:59.569650 2451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:59.569794 2451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:59.570020 2451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:52:59.570365 2451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:52:59.570416 2451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.570447 2451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:52:59.570473 2451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.576215 2451 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:38245
I20260929 18:52:59.576295 2578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:38245 every 8 connection(s)
I20260929 18:52:59.576660 2451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:52:59.579667 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.577513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:54259 (local address 127.29.71.190:35985)
0929 18:52:59.577562 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.577565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.577819 (+ 254us) server_negotiation.cc:407] Connection header received
0929 18:52:59.578335 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.578337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.578342 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.578395 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.579481 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.579482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.579482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.579486 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.579501 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.579504 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.579549 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.579601 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.579613 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:52:59.579861 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.577508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35985 (local address 127.29.71.131:54259)
0929 18:52:59.577698 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.577731 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.578239 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.578455 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.578458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.578465 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.579317 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.579317 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.579346 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.579380 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.579553 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.579555 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.579558 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.579607 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.579655 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260929 18:52:59.580513 2579 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35985
I20260929 18:52:59.580592 2579 heartbeater.cc:464] Registering TS with master...
I20260929 18:52:59.580749 2579 heartbeater.cc:511] Master 127.29.71.190:35985 requested a full tablet report, sending...
I20260929 18:52:59.581139 2128 ts_manager.cc:194] Registered new tserver with Master: 28f8f11648c24672bd3c148621b73079 (127.29.71.131:38245)
I20260929 18:52:59.584472 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2451
I20260929 18:52:59.584596 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:52:59.584796 29982 external_mini_cluster.cc:1428] Setting key 9a93c8e13ba2654379a7ef0a4485b9db
I20260929 18:52:59.585695 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:52:59.588241 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.587298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.29.71.190:35985)
0929 18:52:59.587337 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.587338 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.587378 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:52:59.587457 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.587457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.587461 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.587542 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.588071 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.588072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.588072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.588076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.588088 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.588092 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.588145 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.588180 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.588191 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:52:59.589941 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:52:59.591583 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:52:59.595505 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.593880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:38245 (local address 127.0.0.1:59792)
0929 18:52:59.594128 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.594137 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.594207 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.594460 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.594465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.594473 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.595034 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.595035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.595059 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.595079 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.595339 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.595345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.595348 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.595420 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.595434 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260929 18:52:59.595525 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.594132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.29.71.131:38245)
0929 18:52:59.594324 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.594328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.594337 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:59.594357 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.594359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.594366 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.594462 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.595180 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.595188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.595188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.595192 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.595231 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.595262 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.595314 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.595435 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.595451 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260929 18:52:59.595805 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.593809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:38561 (local address 127.0.0.1:44410)
0929 18:52:59.594079 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.594123 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.594205 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.594967 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.594968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.594971 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.595381 (+ 410us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.595382 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.595388 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.595402 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.595647 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.595647 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.595648 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.595703 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.595717 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":70,"threads_started":1}
I20260929 18:52:59.596086 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.593901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.29.71.129:38561)
0929 18:52:59.594832 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.594835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.594843 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:52:59.594863 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.594865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.594871 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.594964 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.595496 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.595504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.595504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.595509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.595541 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.595569 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.595625 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.595736 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.595753 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":138,"threads_started":1}
I20260929 18:52:59.596617 2513 tablet_service.cc:1467] Processing CreateTablet for tablet fbd0e2fe0af74558ad87d19edf779f23 (DEFAULT_TABLE table=test-table [id=03ca9fe1339b41a0b786098ce402715f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:59.596769 2251 tablet_service.cc:1467] Processing CreateTablet for tablet fbd0e2fe0af74558ad87d19edf779f23 (DEFAULT_TABLE table=test-table [id=03ca9fe1339b41a0b786098ce402715f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:59.597023 2513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd0e2fe0af74558ad87d19edf779f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:59.597158 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd0e2fe0af74558ad87d19edf779f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:59.598557 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.593881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42713 (local address 127.0.0.1:56498)
0929 18:52:59.594287 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.594295 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.594361 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.595644 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.595644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.595647 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.596120 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.596121 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.596127 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.596138 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.598433 (+ 2295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.598434 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.598434 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.598481 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.598493 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
I20260929 18:52:59.599079 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.594132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.29.71.130:42713)
0929 18:52:59.595299 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.595303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.595312 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:59.595338 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.595342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.595352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.595429 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.598261 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.598269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.598269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.598273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.598305 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.598329 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.598394 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.598975 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.598993 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":86,"threads_started":1}
I20260929 18:52:59.599864 2593 tablet_bootstrap.cc:492] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: Bootstrap starting.
I20260929 18:52:59.600174 2594 tablet_bootstrap.cc:492] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Bootstrap starting.
I20260929 18:52:59.600221 2382 tablet_service.cc:1467] Processing CreateTablet for tablet fbd0e2fe0af74558ad87d19edf779f23 (DEFAULT_TABLE table=test-table [id=03ca9fe1339b41a0b786098ce402715f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:52:59.600572 2382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd0e2fe0af74558ad87d19edf779f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:59.600585 2593 tablet_bootstrap.cc:654] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:59.600795 2594 tablet_bootstrap.cc:654] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:59.600917 2593 log.cc:824] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:59.601104 2594 log.cc:824] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:59.601742 2593 tablet_bootstrap.cc:492] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: No bootstrap required, opened a new log
I20260929 18:52:59.601853 2593 ts_tablet_manager.cc:1397] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:52:59.602306 2594 tablet_bootstrap.cc:492] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: No bootstrap required, opened a new log
I20260929 18:52:59.602383 2594 ts_tablet_manager.cc:1397] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:52:59.603832 2597 tablet_bootstrap.cc:492] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Bootstrap starting.
I20260929 18:52:59.604130 2593 raft_consensus.cc:348] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.604315 2593 raft_consensus.cc:374] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:59.604349 2593 raft_consensus.cc:729] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182a7f50f6cf41c9ac25a9759c3ed0bb, State: Initialized, Role: FOLLOWER
I20260929 18:52:59.604549 2597 tablet_bootstrap.cc:654] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:59.604611 2593 consensus_queue.cc:260] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.605005 2593 ts_tablet_manager.cc:1428] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:59.605170 2317 heartbeater.cc:503] Master 127.29.71.190:35985 was elected leader, sending a full tablet report...
I20260929 18:52:59.605237 2597 log.cc:824] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:59.606204 2597 tablet_bootstrap.cc:492] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: No bootstrap required, opened a new log
I20260929 18:52:59.605770 2594 raft_consensus.cc:348] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.606328 2597 ts_tablet_manager.cc:1397] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:52:59.606446 2594 raft_consensus.cc:374] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:59.606506 2594 raft_consensus.cc:729] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f8f11648c24672bd3c148621b73079, State: Initialized, Role: FOLLOWER
I20260929 18:52:59.606633 2594 consensus_queue.cc:260] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.606875 2594 ts_tablet_manager.cc:1428] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 18:52:59.607023 2579 heartbeater.cc:503] Master 127.29.71.190:35985 was elected leader, sending a full tablet report...
I20260929 18:52:59.608738 2597 raft_consensus.cc:348] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.608887 2597 raft_consensus.cc:374] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:59.608940 2597 raft_consensus.cc:729] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a126b43d717041679900e1bdd10fabe4, State: Initialized, Role: FOLLOWER
I20260929 18:52:59.609054 2597 consensus_queue.cc:260] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.609309 2597 ts_tablet_manager.cc:1428] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:52:59.609349 2448 heartbeater.cc:503] Master 127.29.71.190:35985 was elected leader, sending a full tablet report...
I20260929 18:52:59.611346 2601 raft_consensus.cc:482] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:52:59.611420 2601 raft_consensus.cc:504] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.611722 2601 leader_election.cc:288] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182a7f50f6cf41c9ac25a9759c3ed0bb (127.29.71.129:38561), 28f8f11648c24672bd3c148621b73079 (127.29.71.131:38245)
I20260929 18:52:59.612900 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.612016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:38561 (local address 127.29.71.130:42955)
0929 18:52:59.612105 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.612115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.612150 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.612272 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.612272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.612274 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.612661 (+ 387us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.612661 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.612668 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.612677 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.612815 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.612816 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.612816 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.612849 (+ 33us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.612859 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 18:52:59.612916 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.612062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:42955 (local address 127.29.71.129:38561)
0929 18:52:59.612128 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.612130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.612154 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:52:59.612198 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.612199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.612204 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.612268 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.612728 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.612729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.612730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.612733 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.612749 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.612753 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.612819 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.612851 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.612863 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:52:59.613219 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd0e2fe0af74558ad87d19edf779f23" candidate_uuid: "a126b43d717041679900e1bdd10fabe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" is_pre_election: true
I20260929 18:52:59.613406 2271 raft_consensus.cc:2454] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a126b43d717041679900e1bdd10fabe4 in term 0.
I20260929 18:52:59.613663 2338 leader_election.cc:302] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182a7f50f6cf41c9ac25a9759c3ed0bb, a126b43d717041679900e1bdd10fabe4; no voters:
I20260929 18:52:59.613816 2601 raft_consensus.cc:2781] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:52:59.613867 2601 raft_consensus.cc:482] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:52:59.613883 2601 raft_consensus.cc:3037] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:59.614907 2601 raft_consensus.cc:504] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.615118 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.614049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:38245 (local address 127.29.71.130:42789)
0929 18:52:59.614095 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.614104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.614133 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.614374 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.614375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.614377 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:52:59.614771 (+ 394us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:52:59.614772 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:52:59.614778 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:52:59.614790 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:52:59.614992 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:52:59.614992 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:52:59.614993 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 18:52:59.615035 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 18:52:59.615048 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:52:59.615123 2601 leader_election.cc:288] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [CANDIDATE]: Term 1 election: Requested vote from peers 182a7f50f6cf41c9ac25a9759c3ed0bb (127.29.71.129:38561), 28f8f11648c24672bd3c148621b73079 (127.29.71.131:38245)
I20260929 18:52:59.615191 2271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd0e2fe0af74558ad87d19edf779f23" candidate_uuid: "a126b43d717041679900e1bdd10fabe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb"
I20260929 18:52:59.615276 2271 raft_consensus.cc:3037] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:59.615298 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.614150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:42789 (local address 127.29.71.131:38245)
0929 18:52:59.614233 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.614235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.614245 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:52:59.614276 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.614277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.614285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.614355 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.614881 (+ 526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.614883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.614883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.614888 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.614904 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.614908 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.614968 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.615215 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.615229 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 18:52:59.615566 2532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd0e2fe0af74558ad87d19edf779f23" candidate_uuid: "a126b43d717041679900e1bdd10fabe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f8f11648c24672bd3c148621b73079"
I20260929 18:52:59.615659 2532 raft_consensus.cc:3037] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:59.615574 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd0e2fe0af74558ad87d19edf779f23" candidate_uuid: "a126b43d717041679900e1bdd10fabe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f8f11648c24672bd3c148621b73079" is_pre_election: true
I20260929 18:52:59.616330 2271 raft_consensus.cc:2454] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a126b43d717041679900e1bdd10fabe4 in term 1.
I20260929 18:52:59.616434 2532 raft_consensus.cc:2454] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a126b43d717041679900e1bdd10fabe4 in term 1.
I20260929 18:52:59.616569 2338 leader_election.cc:302] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182a7f50f6cf41c9ac25a9759c3ed0bb, a126b43d717041679900e1bdd10fabe4; no voters:
I20260929 18:52:59.616707 2601 raft_consensus.cc:2781] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:59.616890 2601 raft_consensus.cc:686] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 1 LEADER]: Becoming Leader. State: Replica: a126b43d717041679900e1bdd10fabe4, State: Running, Role: LEADER
I20260929 18:52:59.617007 2601 consensus_queue.cc:237] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } }
I20260929 18:52:59.617780 2128 catalog_manager.cc:5495] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a126b43d717041679900e1bdd10fabe4 (127.29.71.130). New cstate: current_term: 1 leader_uuid: "a126b43d717041679900e1bdd10fabe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a126b43d717041679900e1bdd10fabe4" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 } health_report { overall_health: UNKNOWN } } }
I20260929 18:52:59.624433 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.623256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.29.71.130:42713)
0929 18:52:59.623310 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.623313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.623322 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:52:59.623379 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.623379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.623391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.623449 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.624149 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.624150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.624151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.624156 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.624171 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.624175 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.624239 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.624308 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.624323 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:52:59.626314 2532 raft_consensus.cc:1261] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 1 FOLLOWER]: Refusing update from remote peer a126b43d717041679900e1bdd10fabe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:59.626336 2271 raft_consensus.cc:1261] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 1 FOLLOWER]: Refusing update from remote peer a126b43d717041679900e1bdd10fabe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:52:59.626668 2602 consensus_queue.cc:1035] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182a7f50f6cf41c9ac25a9759c3ed0bb" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 38561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:59.626781 2601 consensus_queue.cc:1035] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f8f11648c24672bd3c148621b73079" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:52:59.629619 2605 mvcc.cc:205] Tried to move back new op lower bound from 7334739884546596864 to 7334739884511936512. Current Snapshot: MvccSnapshot[applied={T|T < 7334739884546596864}]
I20260929 18:52:59.629724 2606 mvcc.cc:205] Tried to move back new op lower bound from 7334739884546596864 to 7334739884511936512. Current Snapshot: MvccSnapshot[applied={T|T < 7334739884546596864 or (T in {7334739884546596864})}]
I20260929 18:52:59.631778 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:59.631877 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:52:59.632762 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d35d32f313eb4ea0967df9fa5cd02334: Sending DeleteTablet for 3 replicas of tablet fbd0e2fe0af74558ad87d19edf779f23
I20260929 18:52:59.633136 2382 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd0e2fe0af74558ad87d19edf779f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:59 UTC) from {username='slave'} at 127.0.0.1:56498
I20260929 18:52:59.633195 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd0e2fe0af74558ad87d19edf779f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:59 UTC) from {username='slave'} at 127.0.0.1:44410
I20260929 18:52:59.633405 2616 tablet_replica.cc:330] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: stopping tablet replica
I20260929 18:52:59.633502 2616 raft_consensus.cc:2229] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:59.633505 2513 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd0e2fe0af74558ad87d19edf779f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:52:59 UTC) from {username='slave'} at 127.0.0.1:59792
I20260929 18:52:59.633594 2616 raft_consensus.cc:2258] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:59.633734 2615 tablet_replica.cc:330] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: stopping tablet replica
I20260929 18:52:59.633817 2615 raft_consensus.cc:2229] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:52:59.633848 2617 tablet_replica.cc:330] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: stopping tablet replica
I20260929 18:52:59.633903 2615 raft_consensus.cc:2258] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:59.633917 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2189
I20260929 18:52:59.633949 2617 raft_consensus.cc:2229] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:52:59.634054 2617 raft_consensus.cc:2258] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:52:59.634086 2616 ts_tablet_manager.cc:1905] T fbd0e2fe0af74558ad87d19edf779f23 P 182a7f50f6cf41c9ac25a9759c3ed0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:59.634620 2617 ts_tablet_manager.cc:1905] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:59.634863 2615 ts_tablet_manager.cc:1905] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:52:59.636507 2615 ts_tablet_manager.cc:1918] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:52:59.636598 2615 log.cc:1196] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/fbd0e2fe0af74558ad87d19edf779f23
I20260929 18:52:59.636862 2615 ts_tablet_manager.cc:1939] T fbd0e2fe0af74558ad87d19edf779f23 P a126b43d717041679900e1bdd10fabe4: Deleting consensus metadata
I20260929 18:52:59.636883 2617 ts_tablet_manager.cc:1918] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:52:59.636960 2617 log.cc:1196] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/fbd0e2fe0af74558ad87d19edf779f23
I20260929 18:52:59.637249 2617 ts_tablet_manager.cc:1939] T fbd0e2fe0af74558ad87d19edf779f23 P 28f8f11648c24672bd3c148621b73079: Deleting consensus metadata
I20260929 18:52:59.637279 2114 catalog_manager.cc:4841] TS a126b43d717041679900e1bdd10fabe4 (127.29.71.130:42713): tablet fbd0e2fe0af74558ad87d19edf779f23 (table test-table [id=03ca9fe1339b41a0b786098ce402715f]) successfully deleted
I20260929 18:52:59.637838 2113 catalog_manager.cc:4841] TS 28f8f11648c24672bd3c148621b73079 (127.29.71.131:38245): tablet fbd0e2fe0af74558ad87d19edf779f23 (table test-table [id=03ca9fe1339b41a0b786098ce402715f]) successfully deleted
W20260929 18:52:59.641266 2116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:38561 (error 108)
I20260929 18:52:59.641912 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.641751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:38561 (local address 127.0.0.1:44418)
0929 18:52:59.641821 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.641872 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 18:52:59.641980 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2320
W20260929 18:52:59.642056 2116 catalog_manager.cc:4584] TS 182a7f50f6cf41c9ac25a9759c3ed0bb (127.29.71.129:38561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbd0e2fe0af74558ad87d19edf779f23: Network error: Client connection negotiation failed: client connection to 127.29.71.129:38561: connect: Connection refused (error 111)
I20260929 18:52:59.647702 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2451
I20260929 18:52:59.652072 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2098
2026-09-29T18:52:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (596 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 18:52:59.672711 29982 mini_oidc.cc:152] Starting JWKS server
I20260929 18:52:59.673774 29982 webserver.cc:460] Webserver started at https://127.0.0.1:42595/ using document root <none> and password file <none>
I20260929 18:52:59.673900 29982 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:52:59.674090 29982 webserver.cc:460] Webserver started at http://127.0.0.1:44835/ using document root <none> and password file <none>
2026-09-29T18:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:59Z Disabled control of system clock
I20260929 18:52:59.681623 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:32985
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:32985
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42595/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/testchainca.cert with env {}
W20260929 18:52:59.760800 2624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:59.761015 2624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:59.761039 2624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:59.762125 2624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:59.762173 2624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:59.762187 2624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:59.762200 2624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:59.762218 2624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:52:59.762236 2624 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:52:59.762248 2624 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42595/jwks.json
I20260929 18:52:59.763717 2624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:32985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42595/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:59.763913 2624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:59.764086 2624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:52:59.768852 2624 server_base.cc:1034] running on GCE node
W20260929 18:52:59.768703 2629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.768703 2632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:52:59.768707 2630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:52:59.770223 2624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:52:59.770510 2624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:52:59.771653 2624 hybrid_clock.cc:648] HybridClock initialized: now 1790707979771642 us; error 29 us; skew 500 ppm
I20260929 18:52:59.777896 2624 webserver.cc:460] Webserver started at http://127.29.71.190:42791/ using document root <none> and password file <none>
I20260929 18:52:59.778221 2624 fs_manager.cc:359] Metadata directory not provided
I20260929 18:52:59.778275 2624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:52:59.778435 2624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:52:59.779533 2624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "1a31d474daad444299f4513359c18f85"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "3bb9fa33b9107122e85f662e22b3d277"
server_key_iv: "0d3af639eaf753406259531f318c4c16"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.779907 2624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "1a31d474daad444299f4513359c18f85"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "3bb9fa33b9107122e85f662e22b3d277"
server_key_iv: "0d3af639eaf753406259531f318c4c16"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.781213 2624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:52:59.781922 2638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:52:59.782094 2624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:52:59.782217 2624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "1a31d474daad444299f4513359c18f85"
format_stamp: "Formatted at 2026-09-29 18:52:59 on dist-test-slave-q2kh"
server_key: "3bb9fa33b9107122e85f662e22b3d277"
server_key_iv: "0d3af639eaf753406259531f318c4c16"
server_key_version: "encryptionkey@0"
I20260929 18:52:59.782281 2624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:52:59.792330 2624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:52:59.792627 2624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:52:59.792778 2624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:52:59.797097 2624 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:32985
I20260929 18:52:59.797143 2690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:32985 every 8 connection(s)
I20260929 18:52:59.797487 2624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:52:59.798501 2691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:52:59.800735 2691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85: Bootstrap starting.
I20260929 18:52:59.801272 2691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85: Neither blocks nor log segments found. Creating new log.
I20260929 18:52:59.801506 2691 log.cc:824] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85: Log is configured to *not* fsync() on all Append() calls
I20260929 18:52:59.802146 2691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85: No bootstrap required, opened a new log
I20260929 18:52:59.803709 2691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a31d474daad444299f4513359c18f85" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 32985 } }
I20260929 18:52:59.803848 2691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:52:59.803870 2691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a31d474daad444299f4513359c18f85, State: Initialized, Role: FOLLOWER
I20260929 18:52:59.804010 2691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a31d474daad444299f4513359c18f85" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 32985 } }
I20260929 18:52:59.804092 2691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:52:59.804114 2691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:52:59.804143 2691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:52:59.804602 2691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a31d474daad444299f4513359c18f85" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 32985 } }
I20260929 18:52:59.804683 2691 leader_election.cc:302] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a31d474daad444299f4513359c18f85; no voters:
I20260929 18:52:59.804803 2691 leader_election.cc:288] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:52:59.804952 2694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:52:59.805044 2691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:52:59.805130 2694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [term 1 LEADER]: Becoming Leader. State: Replica: 1a31d474daad444299f4513359c18f85, State: Running, Role: LEADER
I20260929 18:52:59.805235 2694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a31d474daad444299f4513359c18f85" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 32985 } }
I20260929 18:52:59.805644 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2624
I20260929 18:52:59.805763 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:52:59.805984 29982 external_mini_cluster.cc:1428] Setting key 1193d019933a5b08c2754c040899f85d
I20260929 18:52:59.806782 2696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a31d474daad444299f4513359c18f85. Latest consensus state: current_term: 1 leader_uuid: "1a31d474daad444299f4513359c18f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a31d474daad444299f4513359c18f85" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 32985 } } }
I20260929 18:52:59.806795 2695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a31d474daad444299f4513359c18f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a31d474daad444299f4513359c18f85" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 32985 } } }
I20260929 18:52:59.806878 2696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [sys.catalog]: This master's current role is: LEADER
I20260929 18:52:59.806883 2695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85 [sys.catalog]: This master's current role is: LEADER
W20260929 18:52:59.807163 2710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:52:59.807286 2710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:52:59.807367 2713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:52:59.807542 2713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:52:59.810441 2713 catalog_manager.cc:1347] Generated new cluster ID: 3fdaaacbca874afa90b35b7ae448d6ac
I20260929 18:52:59.810497 2713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:52:59.812515 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.806492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48468 (local address 127.29.71.190:32985)
0929 18:52:59.807296 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.807304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.807333 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:52:59.807455 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.807462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.807531 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.807753 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:52:59.807993 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.809308 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.810272 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.810940 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.811076 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:59.811796 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:52:59.811800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:52:59.811800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:52:59.812023 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:52:59.812063 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:52:59.812077 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:52:59.812148 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:52:59.812221 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:52:59.812299 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":73,"threads_started":1}
I20260929 18:52:59.823330 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:52:59.830147 2713 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:52:59.830904 2713 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:52:59.836277 2713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a31d474daad444299f4513359c18f85: Generated new TSK 0
I20260929 18:52:59.836463 2713 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:52:59.863284 2721 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 18:52:59.863406 2721 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260929 18:52:59.863993 2720 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:52:59.846884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:32985 (local address 127.0.0.1:48482)
0929 18:52:59.847121 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 18:52:59.847140 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:52:59.847206 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:52:59.847435 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:52:59.847436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:52:59.847442 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:52:59.847500 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:59.847502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.848455 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.848456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:59.849058 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:52:59.849060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.849555 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.849556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:52:59.849610 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:59.849628 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:52:59.863552 (+ 13924us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:52:59.863659 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.71.190:32985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":123,"threads_started":1}
I20260929 18:52:59.864303 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:52:59.846984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.29.71.190:32985)
0929 18:52:59.847048 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:52:59.847050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:52:59.847203 (+ 153us) server_negotiation.cc:407] Connection header received
0929 18:52:59.847291 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:52:59.847292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:52:59.847314 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:52:59.847402 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:52:59.847575 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.848354 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.849139 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:52:59.849469 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:52:59.849540 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:52:59.849742 (+ 202us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:52:59.863383 (+ 13641us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:52:59.864233 (+ 850us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 18:52:59.864356 2711 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.71.190:32985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 18:52:59.864790 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2624
2026-09-29T18:52:59Z chronyd exiting
I20260929 18:52:59.901101 29982 test_util.cc:175] -----------------------------------------------
I20260929 18:52:59.901228 29982 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790707954762284-29982-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 18:52:59.902318 29982 mini_oidc.cc:152] Starting JWKS server
I20260929 18:52:59.903295 29982 webserver.cc:460] Webserver started at https://127.0.0.1:40863/ using document root <none> and password file <none>
I20260929 18:52:59.903349 29982 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:52:59.903523 29982 webserver.cc:460] Webserver started at http://127.0.0.1:35141/ using document root <none> and password file <none>
2026-09-29T18:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:52:59Z Disabled control of system clock
I20260929 18:52:59.912806 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33639
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33639
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40863/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/testchainca.cert with env {}
W20260929 18:52:59.995005 2727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:52:59.995215 2727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:52:59.995239 2727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:52:59.996428 2727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:52:59.996474 2727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:52:59.996486 2727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:52:59.996498 2727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:52:59.996562 2727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:52:59.996591 2727 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:52:59.996608 2727 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40863/jwks.json
I20260929 18:52:59.998129 2727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40863/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:52:59.998413 2727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:52:59.998632 2727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:53:00.003752 2727 server_base.cc:1034] running on GCE node
W20260929 18:53:00.003717 2733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:00.003767 2732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:00.003717 2735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:00.004124 2727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:00.004427 2727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:00.005591 2727 hybrid_clock.cc:648] HybridClock initialized: now 1790707980005570 us; error 40 us; skew 500 ppm
I20260929 18:53:00.006906 2727 webserver.cc:460] Webserver started at http://127.29.71.190:43785/ using document root <none> and password file <none>
I20260929 18:53:00.007191 2727 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:00.007261 2727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:00.007401 2727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:00.008433 2727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "314e9f36d8154fbab8b29c963d0dd33f"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "1a8344e4f9f5a4aa12b42246487d534a"
server_key_iv: "300f71c5ad3f3f25418b9081f5a0cb04"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.008782 2727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "314e9f36d8154fbab8b29c963d0dd33f"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "1a8344e4f9f5a4aa12b42246487d534a"
server_key_iv: "300f71c5ad3f3f25418b9081f5a0cb04"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.010090 2727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:00.010846 2741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:00.011006 2727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:00.011107 2727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "314e9f36d8154fbab8b29c963d0dd33f"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "1a8344e4f9f5a4aa12b42246487d534a"
server_key_iv: "300f71c5ad3f3f25418b9081f5a0cb04"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.011178 2727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:00.026587 2727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:00.026898 2727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:00.027074 2727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:00.031090 2727 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33639
I20260929 18:53:00.031121 2793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33639 every 8 connection(s)
I20260929 18:53:00.031479 2727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:00.032105 2794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:00.034799 2794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f: Bootstrap starting.
I20260929 18:53:00.035372 2794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:00.035650 2794 log.cc:824] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:00.036343 2794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f: No bootstrap required, opened a new log
I20260929 18:53:00.037334 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2727
I20260929 18:53:00.037456 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:00.037648 29982 external_mini_cluster.cc:1428] Setting key 30a96eced3df8e80389e086c62577960
I20260929 18:53:00.038268 2794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314e9f36d8154fbab8b29c963d0dd33f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33639 } }
I20260929 18:53:00.038471 2794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:00.038520 2794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314e9f36d8154fbab8b29c963d0dd33f, State: Initialized, Role: FOLLOWER
I20260929 18:53:00.038641 2794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314e9f36d8154fbab8b29c963d0dd33f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33639 } }
I20260929 18:53:00.038714 2794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:00.038750 2794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:00.038802 2794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:00.039448 2794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314e9f36d8154fbab8b29c963d0dd33f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33639 } }
I20260929 18:53:00.039536 2794 leader_election.cc:302] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 314e9f36d8154fbab8b29c963d0dd33f; no voters:
I20260929 18:53:00.039709 2794 leader_election.cc:288] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:00.039734 2799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:00.039840 2799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [term 1 LEADER]: Becoming Leader. State: Replica: 314e9f36d8154fbab8b29c963d0dd33f, State: Running, Role: LEADER
I20260929 18:53:00.039927 2799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314e9f36d8154fbab8b29c963d0dd33f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33639 } }
I20260929 18:53:00.039956 2794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:00.040396 2800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 314e9f36d8154fbab8b29c963d0dd33f. Latest consensus state: current_term: 1 leader_uuid: "314e9f36d8154fbab8b29c963d0dd33f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314e9f36d8154fbab8b29c963d0dd33f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33639 } } }
I20260929 18:53:00.040481 2800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:00.040747 2804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:00.041451 2804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:00.041791 2801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "314e9f36d8154fbab8b29c963d0dd33f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314e9f36d8154fbab8b29c963d0dd33f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33639 } } }
I20260929 18:53:00.041923 2801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:00.043331 2804 catalog_manager.cc:1347] Generated new cluster ID: 03c955c1cb8048198f371c7573e4e605
I20260929 18:53:00.043382 2804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:53:00.044112 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:00.038210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.29.71.190:33639)
0929 18:53:00.038434 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:00.038444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:00.038478 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:53:00.038590 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:00.038596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:00.038665 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:00.038818 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:00.039175 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.040799 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.042010 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.042627 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.042760 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.043424 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:00.043428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:00.043428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:00.043633 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:00.043665 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:00.043676 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:00.043754 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:00.043829 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:53:00.043905 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260929 18:53:00.044981 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:53:00.052642 2804 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:00.053164 2804 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:00.057128 2804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 314e9f36d8154fbab8b29c963d0dd33f: Generated new TSK 0
I20260929 18:53:00.057317 2804 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:53:00.082352 2824 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 18:53:00.082441 2824 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:53:00.082795 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:00.068357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.29.71.190:33639)
0929 18:53:00.068426 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:00.068428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:00.068498 (+ 70us) server_negotiation.cc:407] Connection header received
0929 18:53:00.068569 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:00.068570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:00.068595 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:00.068674 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:53:00.068805 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.069606 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.070411 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.070748 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.070821 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.070953 (+ 132us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:53:00.082564 (+ 11611us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:53:00.082722 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 18:53:00.082855 2798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 18:53:00.082835 2823 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:00.068258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33639 (local address 127.0.0.1:40644)
0929 18:53:00.068424 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:00.068435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:00.068496 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:00.068683 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:00.068683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:00.068685 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:53:00.068742 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:00.068744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.069684 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.069685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:00.070336 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:00.070339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.070825 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.070826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:00.070869 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.070873 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:53:00.082676 (+ 11803us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:53:00.082766 (+ 90us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.71.190:33639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260929 18:53:00.083323 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2727
2026-09-29T18:53:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 18:53:00.121104 29982 mini_oidc.cc:152] Starting JWKS server
I20260929 18:53:00.121816 29982 webserver.cc:460] Webserver started at https://127.0.0.1:46521/ using document root <none> and password file <none>
I20260929 18:53:00.121888 29982 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:53:00.122054 29982 webserver.cc:460] Webserver started at http://127.0.0.1:43843/ using document root <none> and password file <none>
2026-09-29T18:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:00Z Disabled control of system clock
I20260929 18:53:00.129623 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33909
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33909
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46521/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:53:00.209070 2830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:00.209311 2830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:00.209345 2830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:00.210513 2830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:00.210580 2830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:00.210600 2830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:00.210620 2830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:00.210649 2830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:53:00.210680 2830 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:53:00.210705 2830 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46521/jwks.json
I20260929 18:53:00.212129 2830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46521/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:00.212395 2830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:00.212671 2830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:00.217511 2836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:00.217515 2838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:00.217530 2835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:00.217694 2830 server_base.cc:1034] running on GCE node
I20260929 18:53:00.217947 2830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:00.218292 2830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:00.219446 2830 hybrid_clock.cc:648] HybridClock initialized: now 1790707980219404 us; error 57 us; skew 500 ppm
I20260929 18:53:00.220628 2830 webserver.cc:460] Webserver started at http://127.29.71.190:43559/ using document root <none> and password file <none>
I20260929 18:53:00.220880 2830 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:00.220928 2830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:00.221083 2830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:00.222203 2830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "15cd5a65946949e4a17e9f2331624f42"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "a63e6d5dd927ad4a6b5c716c66801c28"
server_key_iv: "32b9a273caf4400d7d2f9367109e3023"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.222589 2830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "15cd5a65946949e4a17e9f2331624f42"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "a63e6d5dd927ad4a6b5c716c66801c28"
server_key_iv: "32b9a273caf4400d7d2f9367109e3023"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.223908 2830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:53:00.224575 2844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:00.224756 2830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:00.224844 2830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "15cd5a65946949e4a17e9f2331624f42"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "a63e6d5dd927ad4a6b5c716c66801c28"
server_key_iv: "32b9a273caf4400d7d2f9367109e3023"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.224915 2830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:00.240854 2830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:00.241182 2830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:00.241353 2830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:00.245539 2830 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33909
I20260929 18:53:00.245589 2896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33909 every 8 connection(s)
I20260929 18:53:00.245962 2830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:00.246712 2897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:00.249300 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42: Bootstrap starting.
I20260929 18:53:00.249876 2897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:00.250182 2897 log.cc:824] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:00.250871 2897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42: No bootstrap required, opened a new log
I20260929 18:53:00.252437 2897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd5a65946949e4a17e9f2331624f42" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33909 } }
I20260929 18:53:00.252574 2897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:00.252596 2897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15cd5a65946949e4a17e9f2331624f42, State: Initialized, Role: FOLLOWER
I20260929 18:53:00.252748 2897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd5a65946949e4a17e9f2331624f42" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33909 } }
I20260929 18:53:00.252810 2897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:00.252830 2897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:00.252895 2897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:00.253463 2897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd5a65946949e4a17e9f2331624f42" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33909 } }
I20260929 18:53:00.253572 2897 leader_election.cc:302] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15cd5a65946949e4a17e9f2331624f42; no voters:
I20260929 18:53:00.253749 2897 leader_election.cc:288] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:00.253801 2900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:00.253830 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2830
I20260929 18:53:00.253944 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:00.254014 2900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [term 1 LEADER]: Becoming Leader. State: Replica: 15cd5a65946949e4a17e9f2331624f42, State: Running, Role: LEADER
I20260929 18:53:00.254032 2897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:00.254107 2900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd5a65946949e4a17e9f2331624f42" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33909 } }
I20260929 18:53:00.254217 29982 external_mini_cluster.cc:1428] Setting key 8c144777f30d876041765b464caa3602
I20260929 18:53:00.254835 2902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15cd5a65946949e4a17e9f2331624f42. Latest consensus state: current_term: 1 leader_uuid: "15cd5a65946949e4a17e9f2331624f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd5a65946949e4a17e9f2331624f42" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33909 } } }
I20260929 18:53:00.254920 2902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:00.254514 2901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15cd5a65946949e4a17e9f2331624f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd5a65946949e4a17e9f2331624f42" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33909 } } }
I20260929 18:53:00.255175 2901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:00.255241 2911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:00.255797 2911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:00.257747 2911 catalog_manager.cc:1347] Generated new cluster ID: 52795f32da16437c954ae71ad5091faf
I20260929 18:53:00.257812 2911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:53:00.261299 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:00.254735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.29.71.190:33909)
0929 18:53:00.255407 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:00.255417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:00.255633 (+ 216us) server_negotiation.cc:407] Connection header received
0929 18:53:00.255843 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:00.255851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:00.255939 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:00.256054 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:00.256494 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.257727 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.259062 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.259689 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.259831 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.260565 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:00.260569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:00.260570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:00.260800 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:00.260843 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:00.260857 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:00.260926 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:00.261016 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:53:00.261091 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":67,"threads_started":1}
I20260929 18:53:00.262282 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:53:00.272694 2911 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:00.273186 2911 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:00.282644 2911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15cd5a65946949e4a17e9f2331624f42: Generated new TSK 0
I20260929 18:53:00.282858 2911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:53:00.293212 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:00.289430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.29.71.190:33909)
0929 18:53:00.289480 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:00.289483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:00.289561 (+ 78us) server_negotiation.cc:407] Connection header received
0929 18:53:00.289634 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:00.289634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:00.289647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:00.289719 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:53:00.289813 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.290668 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.291451 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.291774 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.291844 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.292047 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:53:00.293003 (+ 956us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46521/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:53:00.293138 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46521/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":28}
W20260929 18:53:00.293274 2906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60934: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46521/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 18:53:00.295346 2927 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:00.289356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33909 (local address 127.0.0.1:60934)
0929 18:53:00.289491 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:00.289501 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:00.289563 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:00.289723 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:00.289724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:00.289726 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:53:00.289770 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:00.289771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.290742 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.290743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:00.291364 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:00.291366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.291872 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.291872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:00.291925 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.291931 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:53:00.293102 (+ 1171us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46521/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:53:00.295245 (+ 2143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.71.190:33909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46521/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1940,"thread_start_us":90,"threads_started":1}
I20260929 18:53:00.295627 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2830
E20260929 18:53:00.304947 2928 webserver.cc:533] Webserver: sslize failed:
E20260929 18:53:00.305045 2928 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T18:53:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T18:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:00Z Disabled control of system clock
I20260929 18:53:00.340544 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33395
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33395
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/testchainca.cert with env {}
W20260929 18:53:00.419425 2932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:00.419669 2932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:00.419705 2932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:00.420838 2932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:00.420910 2932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:00.420931 2932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:00.420953 2932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:00.420982 2932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:53:00.421005 2932 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:53:00.421029 2932 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 18:53:00.422485 2932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:00.422756 2932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:00.422969 2932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:00.427682 2940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:00.427853 2932 server_base.cc:1034] running on GCE node
W20260929 18:53:00.427690 2937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:00.427704 2938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:00.428210 2932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:00.428491 2932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:00.429642 2932 hybrid_clock.cc:648] HybridClock initialized: now 1790707980429638 us; error 37 us; skew 500 ppm
I20260929 18:53:00.430835 2932 webserver.cc:460] Webserver started at http://127.29.71.190:34401/ using document root <none> and password file <none>
I20260929 18:53:00.431068 2932 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:00.431137 2932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:00.431284 2932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:00.432308 2932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "ba9bd8e5402a4570993389df2f4d9835"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "615e34b15ce8cfdd9ac45fc7fbe8ba34"
server_key_iv: "5189bc5519abd3871cea497492ea9eb9"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.432667 2932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "ba9bd8e5402a4570993389df2f4d9835"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "615e34b15ce8cfdd9ac45fc7fbe8ba34"
server_key_iv: "5189bc5519abd3871cea497492ea9eb9"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.433923 2932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:00.434727 2946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:00.434898 2932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:00.434996 2932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "ba9bd8e5402a4570993389df2f4d9835"
format_stamp: "Formatted at 2026-09-29 18:53:00 on dist-test-slave-q2kh"
server_key: "615e34b15ce8cfdd9ac45fc7fbe8ba34"
server_key_iv: "5189bc5519abd3871cea497492ea9eb9"
server_key_version: "encryptionkey@0"
I20260929 18:53:00.435082 2932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:00.456558 2932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:00.456897 2932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:00.457069 2932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:00.460987 2932 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33395
I20260929 18:53:00.461025 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33395 every 8 connection(s)
I20260929 18:53:00.461392 2932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:00.462455 2999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:00.464653 2999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835: Bootstrap starting.
I20260929 18:53:00.464716 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 2932
I20260929 18:53:00.464818 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:00.465006 29982 external_mini_cluster.cc:1428] Setting key 4b741e9b76c2e5f7b0ee75edd1c2901e
I20260929 18:53:00.465329 2999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:00.465659 2999 log.cc:824] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:00.466403 2999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835: No bootstrap required, opened a new log
I20260929 18:53:00.468160 2999 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9bd8e5402a4570993389df2f4d9835" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33395 } }
I20260929 18:53:00.468299 2999 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:00.468321 2999 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9bd8e5402a4570993389df2f4d9835, State: Initialized, Role: FOLLOWER
I20260929 18:53:00.468464 2999 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9bd8e5402a4570993389df2f4d9835" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33395 } }
I20260929 18:53:00.468571 2999 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:00.468616 2999 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:00.468665 2999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:00.469200 2999 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9bd8e5402a4570993389df2f4d9835" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33395 } }
I20260929 18:53:00.469316 2999 leader_election.cc:302] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba9bd8e5402a4570993389df2f4d9835; no voters:
I20260929 18:53:00.469497 2999 leader_election.cc:288] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:00.469576 3004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:00.469769 2999 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:00.469766 3004 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [term 1 LEADER]: Becoming Leader. State: Replica: ba9bd8e5402a4570993389df2f4d9835, State: Running, Role: LEADER
I20260929 18:53:00.469992 3004 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9bd8e5402a4570993389df2f4d9835" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33395 } }
I20260929 18:53:00.470453 3005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba9bd8e5402a4570993389df2f4d9835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9bd8e5402a4570993389df2f4d9835" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33395 } } }
I20260929 18:53:00.470599 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9bd8e5402a4570993389df2f4d9835. Latest consensus state: current_term: 1 leader_uuid: "ba9bd8e5402a4570993389df2f4d9835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9bd8e5402a4570993389df2f4d9835" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33395 } } }
I20260929 18:53:00.470659 3005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:00.470736 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:00.470682 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:00.465494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.29.71.190:33395)
0929 18:53:00.465747 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:00.465756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:00.465795 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:53:00.466286 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:00.466293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:00.466351 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:00.466520 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:00.466752 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.467618 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.468434 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.468947 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.469076 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.469801 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:00.469804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:00.469805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:00.470033 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:00.470081 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:00.470096 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:00.470199 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:00.470302 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:53:00.470397 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260929 18:53:00.470937 3013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:00.471459 3013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:00.471789 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:53:00.472831 3021 mvcc.cc:205] Tried to move back new op lower bound from 7334739888015106048 to 7334739888005472256. Current Snapshot: MvccSnapshot[applied={T|T < 7334739888015106048}]
I20260929 18:53:00.473258 3013 catalog_manager.cc:1347] Generated new cluster ID: 7239ab023c854bd99670496efdbf73ad
I20260929 18:53:00.473310 3013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:53:00.489701 3013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:00.490458 3013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:00.493654 3013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba9bd8e5402a4570993389df2f4d9835: Generated new TSK 0
I20260929 18:53:00.493820 3013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:53:00.555474 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:00.501138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.29.71.190:33395)
0929 18:53:00.501179 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:00.501181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:00.501269 (+ 88us) server_negotiation.cc:407] Connection header received
0929 18:53:00.501354 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:00.501355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:00.501364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:00.501424 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:53:00.501573 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.502075 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.503216 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.503399 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.503452 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.503630 (+ 178us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:53:00.555196 (+ 51566us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:53:00.555389 (+ 193us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44024: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 18:53:00.555557 3002 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44024: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 18:53:00.555579 3028 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:00.501046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33395 (local address 127.0.0.1:44024)
0929 18:53:00.501204 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:00.501216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:00.501287 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:00.501444 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:00.501445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:00.501447 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:53:00.501508 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:00.501510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.502173 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.502174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:00.503133 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:00.503136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:00.503477 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:00.503478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:00.503537 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:00.503542 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:53:00.555331 (+ 51789us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:53:00.555455 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.71.190:33395: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260929 18:53:00.555756 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 2932
2026-09-29T18:53:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (240 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:53:00 dist-test-slave-q2kh krb5kdc[3040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:53:00 dist-test-slave-q2kh krb5kdc[3040](info): set up 2 sockets
Sep 29 18:53:00 dist-test-slave-q2kh krb5kdc[3040](info): commencing operation
krb5kdc: starting...
W20260929 18:53:01.773552 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 18:53:02.043916 29982 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.188s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:53:02.120689 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33405
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:02.200717 3053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:02.200958 3053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:02.200980 3053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:02.202194 3053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:02.202235 3053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:02.202250 3053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:02.202262 3053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:02.202275 3053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:02.203716 3053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:02.203964 3053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:02.204195 3053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:02.208976 3061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:02.208973 3058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:02.209194 3053 server_base.cc:1034] running on GCE node
W20260929 18:53:02.209292 3059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:02.209542 3053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:02.209829 3053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:02.210968 3053 hybrid_clock.cc:648] HybridClock initialized: now 1790707982210954 us; error 31 us; skew 500 ppm
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.217577 3053 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:53:02.217976 3053 webserver.cc:460] Webserver started at http://127.29.71.190:45953/ using document root <none> and password file <none>
I20260929 18:53:02.218218 3053 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:02.218268 3053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:02.218360 3053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:02.219394 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "c4d6b91d291a4a05b555ccff695b9894"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "7e89aa6b263fbd32b66af49215d5fea2"
server_key_iv: "648c25dd2a4765925db0b04f5f00caa5"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.219705 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "c4d6b91d291a4a05b555ccff695b9894"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "7e89aa6b263fbd32b66af49215d5fea2"
server_key_iv: "648c25dd2a4765925db0b04f5f00caa5"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.220954 3053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:02.221629 3068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.221786 3053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:02.221870 3053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "c4d6b91d291a4a05b555ccff695b9894"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "7e89aa6b263fbd32b66af49215d5fea2"
server_key_iv: "648c25dd2a4765925db0b04f5f00caa5"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.221926 3053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:02.233124 3053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:02.235378 3053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:02.235493 3053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:02.239383 3053 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33405
I20260929 18:53:02.239462 3120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33405 every 8 connection(s)
I20260929 18:53:02.239780 3053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:02.240669 3121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:02.243023 3121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894: Bootstrap starting.
I20260929 18:53:02.243602 3121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:02.243899 3121 log.cc:824] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:02.244565 3121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894: No bootstrap required, opened a new log
I20260929 18:53:02.245198 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3053
I20260929 18:53:02.245281 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:02.245452 29982 external_mini_cluster.cc:1428] Setting key 54a380410c1597189c40deb83fffd488
I20260929 18:53:02.246845 3121 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d6b91d291a4a05b555ccff695b9894" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33405 } }
I20260929 18:53:02.247028 3121 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:02.247069 3121 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d6b91d291a4a05b555ccff695b9894, State: Initialized, Role: FOLLOWER
I20260929 18:53:02.247231 3121 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d6b91d291a4a05b555ccff695b9894" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33405 } }
I20260929 18:53:02.247293 3121 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:02.247315 3121 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:02.247339 3121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:02.248382 3121 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d6b91d291a4a05b555ccff695b9894" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33405 } }
I20260929 18:53:02.248489 3121 leader_election.cc:302] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d6b91d291a4a05b555ccff695b9894; no voters:
I20260929 18:53:02.248632 3121 leader_election.cc:288] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:02.248713 3126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:02.248864 3121 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:02.248883 3126 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [term 1 LEADER]: Becoming Leader. State: Replica: c4d6b91d291a4a05b555ccff695b9894, State: Running, Role: LEADER
I20260929 18:53:02.249116 3126 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d6b91d291a4a05b555ccff695b9894" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33405 } }
I20260929 18:53:02.249538 3128 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d6b91d291a4a05b555ccff695b9894. Latest consensus state: current_term: 1 leader_uuid: "c4d6b91d291a4a05b555ccff695b9894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d6b91d291a4a05b555ccff695b9894" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33405 } } }
I20260929 18:53:02.249637 3128 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:02.250016 3127 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d6b91d291a4a05b555ccff695b9894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d6b91d291a4a05b555ccff695b9894" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33405 } } }
I20260929 18:53:02.250128 3127 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894 [sys.catalog]: This master's current role is: LEADER
W20260929 18:53:02.250814 3142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:53:02.250885 3142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:53:02.250967 3135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:02.251163 3135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:02.252812 3135 catalog_manager.cc:1347] Generated new cluster ID: d1e608b6c9be4fd0aa9f0c388a98daba
I20260929 18:53:02.252871 3135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:53:02.257426 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.246138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.29.71.190:33405)
0929 18:53:02.246344 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.246355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.246389 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:53:02.246803 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.246812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.246875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.247024 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.247828 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.249312 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.250493 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.251080 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.251185 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.253874 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:02.253878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:02.253878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:02.253889 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:02.255784 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.256194 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.256197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.256197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.256313 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.256543 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.256543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.256544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.256816 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:02.256960 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:02.257085 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 18:53:02.257209 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260929 18:53:02.260454 3135 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:02.261062 3135 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:02.262892 3135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d6b91d291a4a05b555ccff695b9894: Generated new TSK 0
I20260929 18:53:02.263056 3135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:53:02.293437 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33405
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:02.375795 3147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:02.376004 3147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:02.376027 3147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:02.377113 3147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:02.377146 3147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:02.377171 3147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:02.378670 3147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:02.378850 3147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:02.379030 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:02.383929 3152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:02.383929 3153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:02.383963 3155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:02.384321 3147 server_base.cc:1034] running on GCE node
I20260929 18:53:02.384521 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:02.384714 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:02.385874 3147 hybrid_clock.cc:648] HybridClock initialized: now 1790707982385867 us; error 39 us; skew 500 ppm
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.388531 3147 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:53:02.389010 3147 webserver.cc:460] Webserver started at http://127.29.71.129:38629/ using document root <none> and password file <none>
I20260929 18:53:02.389233 3147 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:02.389315 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:02.389449 3147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:02.390569 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "af906c7155064959b1e3e9d6add813e1"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "419429ad8075c4f5c9482005f4260220"
server_key_iv: "9788ba4f4f9302fbdb927ee62116fc77"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.390924 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "af906c7155064959b1e3e9d6add813e1"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "419429ad8075c4f5c9482005f4260220"
server_key_iv: "9788ba4f4f9302fbdb927ee62116fc77"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.392198 3147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:53:02.392879 3162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.393045 3147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:02.393144 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "af906c7155064959b1e3e9d6add813e1"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "419429ad8075c4f5c9482005f4260220"
server_key_iv: "9788ba4f4f9302fbdb927ee62116fc77"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.393222 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:02.405179 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:02.407160 3147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:02.407313 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:02.407521 3147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:02.407819 3147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:02.407873 3147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.407915 3147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:02.407943 3147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.413746 3147 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:46841
I20260929 18:53:02.413782 3275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:46841 every 8 connection(s)
I20260929 18:53:02.414134 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:53:02.418448 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3147
I20260929 18:53:02.418560 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:53:02.418766 29982 external_mini_cluster.cc:1428] Setting key 6bbe0387aa5feedfe3620a2fde0c280a
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:53:02.425460 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.415041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:52547 (local address 127.29.71.190:33405)
0929 18:53:02.415117 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.415119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.415335 (+ 216us) server_negotiation.cc:407] Connection header received
0929 18:53:02.416260 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.416262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.416276 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.416340 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.416900 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.417393 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.418499 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.418805 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.418879 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.422474 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:02.422475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:02.422476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:02.422482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:02.423883 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.424392 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.424392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.424393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.424481 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.424796 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.424797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.424797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.424900 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:02.424960 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:02.425310 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 18:53:02.425381 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:53:02.425724 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.415012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33405 (local address 127.29.71.129:52547)
0929 18:53:02.415196 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:02.415238 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:02.416173 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:02.416394 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:02.416398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:02.416649 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:02.416830 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:02.416834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.417504 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.417505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:02.418387 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:02.418389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.418909 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.418910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:02.419095 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.420678 (+ 1583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:02.420686 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:02.422378 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:02.423966 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:02.423973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:02.423975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:02.424314 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:02.424582 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:02.424583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:02.424583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:02.424726 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:02.424967 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:02.424973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:02.425147 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 18:53:02.425268 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:53:02.425447 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260929 18:53:02.426270 3276 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33405
I20260929 18:53:02.426389 3276 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:02.426573 3276 heartbeater.cc:511] Master 127.29.71.190:33405 requested a full tablet report, sending...
I20260929 18:53:02.427115 3085 ts_manager.cc:194] Registered new tserver with Master: af906c7155064959b1e3e9d6add813e1 (127.29.71.129:46841)
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
I20260929 18:53:02.427733 3085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:52547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:53:02.444850 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33405
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:02.524559 3281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:02.524802 3281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:02.524843 3281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:02.526046 3281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:02.526109 3281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:02.526153 3281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:02.527648 3281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:02.527827 3281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:02.527999 3281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:02.532827 3286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:02.532851 3289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:02.532856 3287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:02.533169 3281 server_base.cc:1034] running on GCE node
I20260929 18:53:02.533394 3281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:02.533592 3281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:02.534726 3281 hybrid_clock.cc:648] HybridClock initialized: now 1790707982534714 us; error 27 us; skew 500 ppm
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.537127 3281 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:53:02.537570 3281 webserver.cc:460] Webserver started at http://127.29.71.130:35679/ using document root <none> and password file <none>
I20260929 18:53:02.537751 3281 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:02.537789 3281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:02.537878 3281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:02.538921 3281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "bf034c4a06004e8f97bd5a5f3d4722f4"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "fd0514fb8219c77caec0e5eb5a82593f"
server_key_iv: "d4d4d5906f9be3dc9941c9d8ded2a85a"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.539222 3281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "bf034c4a06004e8f97bd5a5f3d4722f4"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "fd0514fb8219c77caec0e5eb5a82593f"
server_key_iv: "d4d4d5906f9be3dc9941c9d8ded2a85a"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.540424 3281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:53:02.541154 3296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.541348 3281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:02.541426 3281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "bf034c4a06004e8f97bd5a5f3d4722f4"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "fd0514fb8219c77caec0e5eb5a82593f"
server_key_iv: "d4d4d5906f9be3dc9941c9d8ded2a85a"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.541483 3281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:02.553889 3281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:02.555805 3281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:02.555913 3281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:02.556104 3281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:02.556437 3281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:02.556471 3281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.556493 3281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:02.556552 3281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.562784 3281 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:43125
I20260929 18:53:02.562844 3409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:43125 every 8 connection(s)
I20260929 18:53:02.563156 3281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:02.569482 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3281
I20260929 18:53:02.569573 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:53:02.569785 29982 external_mini_cluster.cc:1428] Setting key d72f3ed1a833ed5684eacfc170a87315
I20260929 18:53:02.573346 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.564035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:60699 (local address 127.29.71.190:33405)
0929 18:53:02.564166 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.564168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.564379 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:53:02.565332 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.565334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.565347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.565397 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.566065 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.566632 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.567638 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.567839 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.567926 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.570125 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:02.570127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:02.570128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:02.570135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:02.571805 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.572316 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.572317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.572318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.572384 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.572631 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.572632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.572633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.572780 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:02.572852 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:02.573204 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 18:53:02.573284 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:53:02.573583 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.564005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33405 (local address 127.29.71.130:60699)
0929 18:53:02.564213 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:02.564262 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:02.565217 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:02.565489 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:02.565493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:02.565751 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:02.565937 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:02.565941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.566740 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.566742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:02.567544 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:02.567545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.567934 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.567935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:02.568063 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.568595 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:02.568602 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:02.570033 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:02.571905 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:02.571911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:02.571912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:02.572228 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:02.572458 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:02.572458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:02.572459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:02.572564 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:02.572902 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:02.572907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:02.573066 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 18:53:02.573184 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:53:02.573314 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260929 18:53:02.574028 3410 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33405
I20260929 18:53:02.574113 3410 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:02.574332 3410 heartbeater.cc:511] Master 127.29.71.190:33405 requested a full tablet report, sending...
I20260929 18:53:02.574759 3085 ts_manager.cc:194] Registered new tserver with Master: bf034c4a06004e8f97bd5a5f3d4722f4 (127.29.71.130:43125)
I20260929 18:53:02.575146 3085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:60699
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:53:02.598320 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33405
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:02.679000 3415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:02.679245 3415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:02.679283 3415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:02.680423 3415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:02.680490 3415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:02.680542 3415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:02.682024 3415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:02.682314 3415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:02.682623 3415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:02.687776 3421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:02.687794 3420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:02.687794 3423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:02.688063 3415 server_base.cc:1034] running on GCE node
I20260929 18:53:02.688272 3415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:02.688511 3415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:02.689666 3415 hybrid_clock.cc:648] HybridClock initialized: now 1790707982689655 us; error 26 us; skew 500 ppm
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.692196 3415 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:53:02.692703 3415 webserver.cc:460] Webserver started at http://127.29.71.131:37045/ using document root <none> and password file <none>
I20260929 18:53:02.692888 3415 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:02.692929 3415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:02.693018 3415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:02.694005 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "031fe08561ef4f7385d22e61b2e9f928"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "567b96b1499eec8ca2fd6eed51f25835"
server_key_iv: "735b406ed995354535c297d4b5f44f33"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.694380 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "031fe08561ef4f7385d22e61b2e9f928"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "567b96b1499eec8ca2fd6eed51f25835"
server_key_iv: "735b406ed995354535c297d4b5f44f33"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.695673 3415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:02.696475 3430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.696681 3415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:02.696787 3415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "031fe08561ef4f7385d22e61b2e9f928"
format_stamp: "Formatted at 2026-09-29 18:53:02 on dist-test-slave-q2kh"
server_key: "567b96b1499eec8ca2fd6eed51f25835"
server_key_iv: "735b406ed995354535c297d4b5f44f33"
server_key_version: "encryptionkey@0"
I20260929 18:53:02.696861 3415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:02.713941 3415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:02.716367 3415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:02.716545 3415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:02.716773 3415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:02.717097 3415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:02.717136 3415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.717159 3415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:02.717226 3415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:02.722888 3415 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:33899
I20260929 18:53:02.722973 3543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:33899 every 8 connection(s)
I20260929 18:53:02.723299 3415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:53:02.732613 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.724081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:51365 (local address 127.29.71.190:33405)
0929 18:53:02.724143 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.724145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.724390 (+ 245us) server_negotiation.cc:407] Connection header received
0929 18:53:02.725380 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.725381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.725391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.725458 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.726046 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.726599 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.727587 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.727765 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.727816 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.730099 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:02.730101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:02.730102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:02.730109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:02.731401 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.731778 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.731779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.731779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.731828 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.732046 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.732046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.732047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.732149 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:02.732201 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:02.732476 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:53:02.732546 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:53:02.732913 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.724067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33405 (local address 127.29.71.131:51365)
0929 18:53:02.724250 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:02.724293 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:02.725292 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:02.725517 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:02.725521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:02.725774 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:02.725972 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:02.725976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.726677 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.726678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:02.727512 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:02.727515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.727856 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.727858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:02.727994 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.728606 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:02.728615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:02.730009 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:02.731481 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:02.731488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:02.731489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:02.731713 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:02.731901 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:02.731901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:02.731902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:02.731991 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:02.732221 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:02.732226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:02.732343 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:53:02.732449 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:53:02.732571 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260929 18:53:02.733091 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3415
I20260929 18:53:02.733193 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:53:02.733394 29982 external_mini_cluster.cc:1428] Setting key 7c51bc9b63b4c6a688d744c77bd8721f
I20260929 18:53:02.733530 3544 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33405
I20260929 18:53:02.733656 3544 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:02.733853 3544 heartbeater.cc:511] Master 127.29.71.190:33405 requested a full tablet report, sending...
I20260929 18:53:02.734285 3084 ts_manager.cc:194] Registered new tserver with Master: 031fe08561ef4f7385d22e61b2e9f928 (127.29.71.131:33899)
I20260929 18:53:02.734694 3084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:51365
I20260929 18:53:02.745431 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/test-admin.keytab.
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.763569 29982 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 18:53:02.765491 3551 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:02.764553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.29.71.129:46841)
0929 18:53:02.764727 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.764733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.764753 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:53:02.764829 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.764832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.764848 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.764934 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.765379 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52976: BlockingRecv error: recv got EOF from 127.0.0.1:52976 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260929 18:53:02.765580 29982 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.766660 29982 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.767647 29982 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:53:02.776093 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.768281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.29.71.129:46841)
0929 18:53:02.768338 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.768340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.768349 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:02.768419 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.768420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.768432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.768505 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.768906 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.769647 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.770499 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.770836 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.770946 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.773282 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:02.773292 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:02.773292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:02.773308 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:02.774849 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.775260 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.775263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.775264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.775322 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.775522 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.775523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.775523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.775666 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:02.775842 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:02.775897 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 18:53:02.775976 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 18:53:02.778010 3551 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:02.777421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.29.71.129:46841)
0929 18:53:02.777469 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.777471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.777508 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:53:02.777579 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.777579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.777590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.777645 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.777966 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52996: BlockingRecv error: recv got EOF from 127.0.0.1:52996 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260929 18:53:02.778214 29982 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.779251 29982 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.780186 29982 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:53:02.787832 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.780732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.29.71.129:46841)
0929 18:53:02.780782 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.780783 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.780792 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:02.780852 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.780853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.780870 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.780914 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.781265 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.781744 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.782497 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.782696 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.782760 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.784777 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:02.784778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:02.784778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:02.784784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:02.786387 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.786713 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.786714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.786714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.786764 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.787151 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.787152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.787152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.787279 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:02.787382 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:02.787516 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:53:02.787590 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 18:53:02.789366 3551 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:02.788899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53024 (local address 127.29.71.129:46841)
0929 18:53:02.788936 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.788938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.788963 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:53:02.789028 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.789028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.789037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.789094 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.789306 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53024: BlockingRecv error: recv got EOF from 127.0.0.1:53024 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 18:53:02.789523 29982 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790707954762284-29982-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.790606 29982 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:02.791623 29982 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.129@KRBTEST.COM
I20260929 18:53:02.799028 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:02.792241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.29.71.129:46841)
0929 18:53:02.792278 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:02.792280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:02.792364 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:53:02.792441 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:02.792442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:02.792451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:02.792526 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:02.792959 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.793431 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.794151 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:02.794387 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:02.794496 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:02.796807 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:02.796809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:02.796809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:02.796815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:02.797967 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.798409 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.798410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.798411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.798461 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:02.798625 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:02.798626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:02.798626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:02.798706 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:02.798784 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:02.798904 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 18:53:02.798959 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:53:02.799769 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3147
I20260929 18:53:02.804493 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3281
I20260929 18:53:02.809002 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3415
I20260929 18:53:02.813228 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3053
2026-09-29T18:53:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2083 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3564](info): set up 2 sockets
Sep 29 18:53:02 dist-test-slave-q2kh krb5kdc[3564](info): commencing operation
krb5kdc: starting...
W20260929 18:53:03.858451 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:53:03 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab.
Entry for principal oryx/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab.
I20260929 18:53:03.979046 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33865
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab
--principal=oryx/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:04.061030 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:04.061239 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:04.061259 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:04.062453 3577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:04.062501 3577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:04.062516 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:04.062528 3577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:04.062541 3577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:04.064200 3577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab
--principal=oryx/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:04.064455 3577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:04.064705 3577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:04.069594 3583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.069635 3585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.069624 3582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:04.069739 3577 server_base.cc:1034] running on GCE node
I20260929 18:53:04.070125 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:04.070451 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:04.071609 3577 hybrid_clock.cc:648] HybridClock initialized: now 1790707984071594 us; error 34 us; skew 500 ppm
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:04.074097 3577 init.cc:377] Logged in from keytab as oryx/127.29.71.190@KRBTEST.COM (short username oryx)
I20260929 18:53:04.074646 3577 webserver.cc:460] Webserver started at http://127.29.71.190:40263/ using document root <none> and password file <none>
I20260929 18:53:04.074878 3577 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:04.074922 3577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:04.075079 3577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:04.076177 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "b4bfd04aeec54be189cc56e88fce86a2"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "78829b768e6d40af6092fd89bb9fe059"
server_key_iv: "bffc764a91e62d66539e0148a3f3a2dc"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.076561 3577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "b4bfd04aeec54be189cc56e88fce86a2"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "78829b768e6d40af6092fd89bb9fe059"
server_key_iv: "bffc764a91e62d66539e0148a3f3a2dc"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.077872 3577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:04.078671 3592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.078866 3577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.078938 3577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "b4bfd04aeec54be189cc56e88fce86a2"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "78829b768e6d40af6092fd89bb9fe059"
server_key_iv: "bffc764a91e62d66539e0148a3f3a2dc"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.079022 3577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:04.101146 3577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:04.103344 3577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:04.103492 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:04.107755 3577 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33865
I20260929 18:53:04.107839 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33865 every 8 connection(s)
I20260929 18:53:04.108198 3577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:04.109211 3645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:04.111578 3645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: Bootstrap starting.
I20260929 18:53:04.112178 3645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:04.112468 3645 log.cc:824] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:04.113202 3645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: No bootstrap required, opened a new log
I20260929 18:53:04.114210 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3577
I20260929 18:53:04.114311 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:04.114507 29982 external_mini_cluster.cc:1428] Setting key 52a8b15ca4476a854ab8d7a391b5ca73
I20260929 18:53:04.115173 3645 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } }
I20260929 18:53:04.115361 3645 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:04.115411 3645 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bfd04aeec54be189cc56e88fce86a2, State: Initialized, Role: FOLLOWER
I20260929 18:53:04.115542 3645 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } }
I20260929 18:53:04.115623 3645 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:04.115681 3645 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:04.115739 3645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:04.116626 3645 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } }
I20260929 18:53:04.116751 3645 leader_election.cc:302] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4bfd04aeec54be189cc56e88fce86a2; no voters:
I20260929 18:53:04.116989 3645 leader_election.cc:288] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:04.117000 3650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:04.117137 3650 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 1 LEADER]: Becoming Leader. State: Replica: b4bfd04aeec54be189cc56e88fce86a2, State: Running, Role: LEADER
I20260929 18:53:04.117223 3650 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } }
I20260929 18:53:04.117265 3645 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:04.117547 3650 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } }
I20260929 18:53:04.117707 3650 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:04.117769 3652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bfd04aeec54be189cc56e88fce86a2. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } }
I20260929 18:53:04.117940 3652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:04.118127 3657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:04.118659 3657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:04.120746 3657 catalog_manager.cc:1347] Generated new cluster ID: f8e56dbb12c3427aa58df443e60df5d2
I20260929 18:53:04.120803 3657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:04.125890 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.115086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.29.71.190:33865)
0929 18:53:04.115276 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.115287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.115325 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:53:04.115469 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.115477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.115536 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.115694 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.116361 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.117626 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.118725 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.119325 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.119432 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.122196 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.122200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.122200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.122212 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.124126 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.124486 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.124489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.124489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.124598 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.124763 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.124763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.124763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.125038 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.125164 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.125543 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.125668 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260929 18:53:04.132195 3657 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:04.132676 3657 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:04.144223 3657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: Generated new TSK 0
I20260929 18:53:04.144424 3657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab.
Entry for principal oryx/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab.
I20260929 18:53:04.174140 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33865
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab
--principal=oryx/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:04.260363 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:04.260591 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:04.260618 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:04.261821 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:04.261873 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:04.261902 3671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:04.263603 3671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab
--principal=oryx/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:04.263895 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:04.264164 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:04.269573 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.269516 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.269587 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:04.269979 3671 server_base.cc:1034] running on GCE node
I20260929 18:53:04.270298 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:04.270512 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:04.271669 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790707984271657 us; error 26 us; skew 500 ppm
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:04.274299 3671 init.cc:377] Logged in from keytab as oryx/127.29.71.129@KRBTEST.COM (short username oryx)
I20260929 18:53:04.274751 3671 webserver.cc:460] Webserver started at http://127.29.71.129:45623/ using document root <none> and password file <none>
I20260929 18:53:04.274948 3671 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:04.274986 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:04.275079 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:04.276078 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "0d1f56ca3d184435b0e9a08383d25abf"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "95082266dc164a1bd1deba53a4c8d3be"
server_key_iv: "3f16a65909cadcefca80d7573a88e961"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.276386 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "0d1f56ca3d184435b0e9a08383d25abf"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "95082266dc164a1bd1deba53a4c8d3be"
server_key_iv: "3f16a65909cadcefca80d7573a88e961"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.277763 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:04.278673 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.278916 3671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:53:04.279003 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "0d1f56ca3d184435b0e9a08383d25abf"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "95082266dc164a1bd1deba53a4c8d3be"
server_key_iv: "3f16a65909cadcefca80d7573a88e961"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.279064 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:04.302518 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:04.304673 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:04.304786 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:04.305022 3671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:04.305361 3671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:04.305397 3671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.305420 3671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:04.305473 3671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.311419 3671 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:34961
I20260929 18:53:04.311470 3799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:34961 every 8 connection(s)
I20260929 18:53:04.311780 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:04.319428 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3671
I20260929 18:53:04.319556 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:53:04.319787 29982 external_mini_cluster.cc:1428] Setting key bf22084cf63c6031fbf490798ee2f994
I20260929 18:53:04.322191 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.312478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:45805 (local address 127.29.71.190:33865)
0929 18:53:04.312547 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.312549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.312730 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:53:04.313222 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.313223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.313234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.313296 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.314386 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.314909 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.315881 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.316082 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.316144 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.318822 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.318823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.318823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.318830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.320661 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.321115 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.321117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.321117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.321194 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.321461 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.321462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.321462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.321591 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.321675 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.321973 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.322064 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:53:04.322386 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.312430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:45805)
0929 18:53:04.312596 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.312636 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.313134 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.313405 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.313409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.313979 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.314256 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.314262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.314989 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.314990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.315784 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.315786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.316181 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.316182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.316474 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.317110 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.317118 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.318714 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.320757 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.320765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.320767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.321015 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.321268 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.321269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.321269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.321385 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.321665 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.321671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.321819 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.321936 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.322103 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260929 18:53:04.322897 3800 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33865
I20260929 18:53:04.323005 3800 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:04.323222 3800 heartbeater.cc:511] Master 127.29.71.190:33865 requested a full tablet report, sending...
I20260929 18:53:04.323743 3609 ts_manager.cc:194] Registered new tserver with Master: 0d1f56ca3d184435b0e9a08383d25abf (127.29.71.129:34961)
I20260929 18:53:04.324368 3609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:45805
WARNING: no policy specified for oryx/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab.
Entry for principal oryx/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab.
I20260929 18:53:04.346424 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33865
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab
--principal=oryx/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:04.427341 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:04.427531 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:04.427553 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:04.428681 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:04.428722 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:04.428748 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:04.430244 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab
--principal=oryx/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:04.430475 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:04.430714 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:04.435724 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.435725 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.435950 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:04.435942 3805 server_base.cc:1034] running on GCE node
I20260929 18:53:04.436237 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:04.436470 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:04.437702 3805 hybrid_clock.cc:648] HybridClock initialized: now 1790707984437686 us; error 25 us; skew 500 ppm
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:04.440220 3805 init.cc:377] Logged in from keytab as oryx/127.29.71.130@KRBTEST.COM (short username oryx)
I20260929 18:53:04.440666 3805 webserver.cc:460] Webserver started at http://127.29.71.130:45723/ using document root <none> and password file <none>
I20260929 18:53:04.440894 3805 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:04.440944 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:04.441108 3805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:04.442122 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "ed16a57445e028e438b7f8a044f078cb"
server_key_iv: "a720aa73448177f736833b4d2b6dc4d6"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.442526 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "ed16a57445e028e438b7f8a044f078cb"
server_key_iv: "a720aa73448177f736833b4d2b6dc4d6"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.443771 3805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:04.444621 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.444911 3805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:04.445053 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "ed16a57445e028e438b7f8a044f078cb"
server_key_iv: "a720aa73448177f736833b4d2b6dc4d6"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.445169 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:04.468297 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:04.470909 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:04.471100 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:04.471312 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:04.471663 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:04.471727 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.471776 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:04.471804 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.478420 3805 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:33565
I20260929 18:53:04.478468 3933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:33565 every 8 connection(s)
I20260929 18:53:04.478808 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:04.481366 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3805
I20260929 18:53:04.481498 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:53:04.481705 29982 external_mini_cluster.cc:1428] Setting key c73c8f5e6fca02ce129dd28a6eda52e1
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:04.489513 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.479550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:37505 (local address 127.29.71.190:33865)
0929 18:53:04.479624 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.479627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.479905 (+ 278us) server_negotiation.cc:407] Connection header received
0929 18:53:04.480436 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.480438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.480450 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.480513 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.481682 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.482519 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.483731 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.483945 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.484006 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.486680 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.486682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.486682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.486690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.488120 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.488540 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.488541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.488542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.488611 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.488872 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.488873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.488874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.488992 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.489071 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.489338 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.489420 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:53:04.489724 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.479539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:37505)
0929 18:53:04.479711 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.479757 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.480314 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.480590 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.480594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.481289 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.481551 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.481558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.482628 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.482629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.483621 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.483625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.484033 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.484034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.484206 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.484759 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.484770 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.486577 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.488222 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.488229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.488231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.488463 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.488680 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.488681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.488681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.488803 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.489060 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.489067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.489208 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.489316 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.489475 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for oryx/127.29.71.131@KRBTEST.COM; defaulting to no policy
I20260929 18:53:04.490294 3934 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33865
I20260929 18:53:04.490388 3934 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:04.490585 3934 heartbeater.cc:511] Master 127.29.71.190:33865 requested a full tablet report, sending...
I20260929 18:53:04.490955 3609 ts_manager.cc:194] Registered new tserver with Master: 27a0da3e66f34b39a0a7a0fbbf0bd8c5 (127.29.71.130:33565)
I20260929 18:53:04.491282 3609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:37505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab.
Entry for principal oryx/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab.
I20260929 18:53:04.509388 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33865
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab
--principal=oryx/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:04.591001 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:04.591264 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:04.591303 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:04.592532 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:04.592608 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:04.592654 3939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:04.594215 3939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab
--principal=oryx/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:04.594508 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:04.594744 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:53:04.600092 3939 server_base.cc:1034] running on GCE node
W20260929 18:53:04.600045 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.600047 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.600059 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:04.600518 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:04.600734 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:04.601895 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790707984601873 us; error 36 us; skew 500 ppm
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:04.604944 3939 init.cc:377] Logged in from keytab as oryx/127.29.71.131@KRBTEST.COM (short username oryx)
I20260929 18:53:04.605438 3939 webserver.cc:460] Webserver started at http://127.29.71.131:37105/ using document root <none> and password file <none>
I20260929 18:53:04.605692 3939 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:04.605738 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:04.605891 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:04.607033 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "52270bceeff84e628caf71897b7e242c"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "dedaf90ff2d9875bf6a2ac363cec9e7d"
server_key_iv: "9927be6b5cf0f9283fa5c5c4c3b5ba54"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.607412 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "52270bceeff84e628caf71897b7e242c"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "dedaf90ff2d9875bf6a2ac363cec9e7d"
server_key_iv: "9927be6b5cf0f9283fa5c5c4c3b5ba54"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.608816 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:53:04.609623 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.609812 3939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:53:04.609913 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "52270bceeff84e628caf71897b7e242c"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "dedaf90ff2d9875bf6a2ac363cec9e7d"
server_key_iv: "9927be6b5cf0f9283fa5c5c4c3b5ba54"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.610005 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:04.657028 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:04.659260 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:04.659435 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:04.659682 3939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:04.660013 3939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:04.660068 3939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.660113 3939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:04.660138 3939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.666066 3939 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:44429
I20260929 18:53:04.666188 4067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:44429 every 8 connection(s)
I20260929 18:53:04.666512 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:04.675071 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 3939
I20260929 18:53:04.675187 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:53:04.675386 29982 external_mini_cluster.cc:1428] Setting key f4f0d325d8f3ad71dc88861c16c6b457
I20260929 18:53:04.676496 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.667215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:54953 (local address 127.29.71.190:33865)
0929 18:53:04.667282 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.667284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.667517 (+ 233us) server_negotiation.cc:407] Connection header received
0929 18:53:04.667987 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.667988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.667999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.668063 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.669001 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.669490 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.670733 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.670923 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.670987 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.673232 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.673233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.673234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.673240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.674721 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.675293 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.675294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.675295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.675362 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.675715 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.675716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.675717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.675887 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.675964 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.676300 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.676415 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:53:04.676734 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.667201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:54953)
0929 18:53:04.667367 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.667411 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.667893 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.668125 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.668128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.668684 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.668919 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.668924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.669588 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.669590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.670646 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.670649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.671006 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.671007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.671127 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.671612 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.671621 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.673115 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.674798 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.674804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.674806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.675190 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.675455 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.675456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.675457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.675622 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.675962 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.675968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.676153 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.676257 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.676476 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260929 18:53:04.677240 4068 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33865
I20260929 18:53:04.677349 4068 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:04.677533 4068 heartbeater.cc:511] Master 127.29.71.190:33865 requested a full tablet report, sending...
I20260929 18:53:04.677969 3609 ts_manager.cc:194] Registered new tserver with Master: 52270bceeff84e628caf71897b7e242c (127.29.71.131:44429)
I20260929 18:53:04.678419 3609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:54953
I20260929 18:53:04.687356 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.71.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.189.keytab.
Entry for principal oryx/127.29.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.189.keytab.
I20260929 18:53:04.714509 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:43121
--webserver_interface=127.29.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.189.keytab
--principal=oryx/127.29.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33865,127.29.71.189:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:04.799427 4073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:04.799666 4073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:04.799703 4073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:04.800871 4073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:04.800935 4073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:04.800962 4073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:04.800994 4073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:04.801016 4073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:04.802556 4073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33865,127.29.71.189:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.189.keytab
--principal=oryx/127.29.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
W20260929 18:53:04.802706 4073 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.71.190:33865,127.29.71.189:43121'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 18:53:04.802860 4073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:04.803082 4073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:04.807767 4079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.807796 4078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.807951 4081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:04.807893 4073 server_base.cc:1034] running on GCE node
I20260929 18:53:04.808249 4073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:04.808568 4073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:04.809741 4073 hybrid_clock.cc:648] HybridClock initialized: now 1790707984809730 us; error 40 us; skew 500 ppm
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:04.812431 4073 init.cc:377] Logged in from keytab as oryx/127.29.71.189@KRBTEST.COM (short username oryx)
I20260929 18:53:04.813133 4073 webserver.cc:460] Webserver started at http://127.29.71.189:34127/ using document root <none> and password file <none>
I20260929 18:53:04.813407 4073 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:04.813483 4073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:04.813637 4073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:04.814945 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/instance:
uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "e39a7af23417df8b6aed19e6c662d1c5"
server_key_iv: "871486dff4917d5ab9b5bc1225efcf6d"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.815323 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal/instance:
uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "e39a7af23417df8b6aed19e6c662d1c5"
server_key_iv: "871486dff4917d5ab9b5bc1225efcf6d"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.816740 4073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:53:04.817543 4088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.817776 4073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:04.817901 4073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "e39a7af23417df8b6aed19e6c662d1c5"
server_key_iv: "871486dff4917d5ab9b5bc1225efcf6d"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.817991 4073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:04.837564 4073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:04.839990 4073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:04.840118 4073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:04.843849 4073 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.189:43121
I20260929 18:53:04.843953 4140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.189:43121 every 8 connection(s)
I20260929 18:53:04.844269 4073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
I20260929 18:53:04.845309 4141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:04.846942 4141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:53:04.849794 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 4073
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.189@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.71.189@KRBTEST.COM
I20260929 18:53:04.857581 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.847832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.29.71.190:33865)
0929 18:53:04.847902 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.847904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.848164 (+ 260us) server_negotiation.cc:407] Connection header received
0929 18:53:04.848729 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.848731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.848749 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.848803 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.849448 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.850227 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.851781 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.851993 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.852079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.854519 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.854521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.854521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.854529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.855962 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.856383 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.856385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.856385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.856452 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.856747 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.856748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.856749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.856991 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.857069 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.857447 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.857516 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:53:04.857862 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.847873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:41762)
0929 18:53:04.848040 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.848076 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.848545 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.848885 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.848888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.849142 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.849339 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.849344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.850308 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.850309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.851690 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.851693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.852086 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.852087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.852239 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.852761 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.852771 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.854427 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.856076 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.856082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.856084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.856304 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.856537 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.856538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.856539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.856669 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.857084 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.857090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.857262 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.857411 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.857626 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260929 18:53:04.858630 4141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:53:04.859580 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.850461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.29.71.189:43121)
0929 18:53:04.850624 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.850629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.850671 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:53:04.850694 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.850696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.850712 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.850804 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.851479 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.852181 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.853226 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.853527 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.853634 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.856465 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.856469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.856469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.856474 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.858216 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.858634 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.858637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.858638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.858720 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.858931 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.858932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.858933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.859112 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.859258 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.859360 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.859467 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.189@KRBTEST.COM for oryx/127.29.71.189@KRBTEST.COM
I20260929 18:53:04.865963 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.858945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.0.0.1:38202)
0929 18:53:04.858988 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.858998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.859036 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.859259 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.859260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.859483 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.859538 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.859540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.860287 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.860288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.861139 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.861142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.861501 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.861502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.861554 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.861862 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.861866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.863021 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.865180 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.865181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.865182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.865336 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.865502 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.865503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.865503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.865549 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.865713 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.865714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.865743 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.865792 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.865901 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:53:04.866071 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.858982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.29.71.189:43121)
0929 18:53:04.859136 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.859137 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.859144 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:53:04.859163 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.859164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.859175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.859240 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.859598 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.860205 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.861225 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.861450 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.861502 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.863094 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.863095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.863096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.863101 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.865113 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.865399 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.865400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.865400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.865447 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.865586 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.865587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.865587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.865667 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.865710 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.865805 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.865947 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260929 18:53:04.867357 4141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Bootstrap starting.
I20260929 18:53:04.867987 4141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:04.868256 4141 log.cc:824] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:04.868999 4141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: No bootstrap required, opened a new log
I20260929 18:53:04.870458 4141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } }
I20260929 18:53:04.870611 4141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:04.870672 4141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1135f0b36f4ef2a7b100e4de7a9b8f, State: Initialized, Role: FOLLOWER
I20260929 18:53:04.870821 4141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } }
I20260929 18:53:04.871124 4141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:04.871094 4148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } } }
I20260929 18:53:04.871212 4148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:53:04.872776 4159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:53:04.872833 4159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:53:04.874390 4073 master_runner.cc:428] Detected that this master 1a1135f0b36f4ef2a7b100e4de7a9b8f is joining an existing cluster
I20260929 18:53:04.874439 4073 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.71.189:43121
I20260929 18:53:04.882530 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.875475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.29.71.190:33865)
0929 18:53:04.875556 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.875558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.875619 (+ 61us) server_negotiation.cc:407] Connection header received
0929 18:53:04.875694 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.875696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.875708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.875771 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.876194 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.876717 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.877761 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.878062 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.878141 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.879398 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.879400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.879400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.879407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.881011 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.881460 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.881463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.881463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.881526 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.881891 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.881892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.881892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.882042 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.882090 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.882353 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.882435 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:53:04.882885 4166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.875390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:41768)
0929 18:53:04.875562 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.875573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.875634 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.875791 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.875793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.876017 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.876104 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.876107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.876799 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.876801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.877671 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.877673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.878178 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.878180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.878234 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.878885 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.878889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.879306 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.881129 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.881130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.881130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.881386 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.881628 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.881629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.881630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.881713 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.882140 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.882141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.882207 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.882676 (+ 469us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.882776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260929 18:53:04.883167 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.875417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.0.0.1:38218)
0929 18:53:04.875723 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.875732 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.875772 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.875911 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.875912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.876103 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.876166 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.876168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.877045 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.877045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.877826 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.877827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.878257 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.878258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.878323 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.878760 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.878766 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.879306 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.881900 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.881901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.881901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.882074 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.882315 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.882316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.882316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.882373 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.882829 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.882830 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.882861 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.882941 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.883048 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":85,"threads_started":1}
I20260929 18:53:04.883177 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.875512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.29.71.189:43121)
0929 18:53:04.875587 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.875590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.875818 (+ 228us) server_negotiation.cc:407] Connection header received
0929 18:53:04.875832 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.875832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.875839 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.875884 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.876230 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.876961 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.877910 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.878098 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.878211 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.879664 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.879665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.879666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.879672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.881816 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.882152 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.882185 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.882185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.882239 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.882436 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.882437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.882438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.882544 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.882618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.882962 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.883098 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":60}
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.190@KRBTEST.COM for oryx/127.29.71.189@KRBTEST.COM
I20260929 18:53:04.893029 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.885379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.0.0.1:38234)
0929 18:53:04.885499 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.885541 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.885630 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.885767 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.885769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.886342 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.886398 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.886400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.887342 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.887343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.887925 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.887927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.888382 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.888383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.888444 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.888935 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.888939 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.890439 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.892050 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.892056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.892057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.892248 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.892444 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.892445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.892445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.892502 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.892758 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.892764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.892804 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.892870 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.892957 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260929 18:53:04.893087 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.885451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.29.71.189:43121)
0929 18:53:04.885490 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.885491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.885629 (+ 138us) server_negotiation.cc:407] Connection header received
0929 18:53:04.885687 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.885688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.885698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.885764 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.886487 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.887252 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.888010 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.888307 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.888378 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.890523 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.890524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.890524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.890540 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.891960 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.892318 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.892319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.892320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.892377 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.892564 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.892564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.892565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.892695 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.892762 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.892910 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.892994 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:53:04.893684 3609 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.71.189:43121: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f"
member_type: NON_VOTER
last_known_addr {
host: "127.29.71.189"
port: 43121
}
attrs {
promote: true
}
}
dest_uuid: "b4bfd04aeec54be189cc56e88fce86a2"
cas_config_opid_index: -1
I20260929 18:53:04.894033 3609 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } }
I20260929 18:53:04.894567 4168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1a1135f0b36f4ef2a7b100e4de7a9b8f (127.29.71.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:04.894905 4115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:04.895700 4115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Refusing update from remote peer b4bfd04aeec54be189cc56e88fce86a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 18:53:04.896056 4169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:04.896682 4168 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.71.189:43121
I20260929 18:53:04.896770 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:04.896863 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:04.897193 4073 master.cc:537] Master@127.29.71.189:43121 shutting down...
I20260929 18:53:04.899051 4115 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1a1135f0b36f4ef2a7b100e4de7a9b8f (127.29.71.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:04.900101 4177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:04.900187 4177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: LEARNER
I20260929 18:53:04.900226 4177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:04.900273 4177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: LEARNER
I20260929 18:53:04.900311 4148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bfd04aeec54be189cc56e88fce86a2. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:04.900537 4148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: LEARNER
I20260929 18:53:04.901594 4073 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Raft consensus shutting down.
I20260929 18:53:04.901724 4073 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Raft consensus is shut down!
I20260929 18:53:04.901774 4073 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: stopping tablet replica
I20260929 18:53:04.914918 4073 master.cc:552] Master@127.29.71.189:43121 shutdown complete.
I20260929 18:53:04.914986 4073 master_runner.cc:305] Clearing existing system tablet
I20260929 18:53:04.915122 4073 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:04.915171 4073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:04.915937 4073 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:53:04.916554 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.916707 4073 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:04.916772 4073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:04.916798 4073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "e39a7af23417df8b6aed19e6c662d1c5"
server_key_iv: "871486dff4917d5ab9b5bc1225efcf6d"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.917654 4073 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:04.919427 4073 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:53:04.919498 4073 log.cc:1196] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 18:53:04.919726 4073 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Deleting consensus metadata
I20260929 18:53:04.919855 4073 master_runner.cc:315] Copying system tablet from 127.29.71.190:33865
I20260929 18:53:04.920449 4073 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.71.190:33865
I20260929 18:53:04.927228 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.920738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:41784)
0929 18:53:04.920932 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:04.920943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:04.921011 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:04.921161 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:04.921161 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:04.921346 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:04.921403 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.921405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.922433 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.922434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.922965 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:04.922967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.923440 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.923441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:04.923499 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.924016 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:04.924026 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:04.924342 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:04.926255 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.926256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.926256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.926448 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.926650 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:04.926650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:04.926651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:04.926707 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:04.926974 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:04.926975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:04.927013 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:53:04.927068 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:53:04.927147 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":141,"threads_started":1}
I20260929 18:53:04.927289 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.920836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.29.71.190:33865)
0929 18:53:04.920903 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.920905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.921017 (+ 112us) server_negotiation.cc:407] Connection header received
0929 18:53:04.921055 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.921056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.921068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.921140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.921473 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.922356 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.923046 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.923351 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.923423 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.924438 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.924440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.924441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.924448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.926131 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.926518 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.926519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.926519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.926581 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.926769 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.926770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.926770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.926910 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.926980 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.927109 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.927188 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:53:04.927942 3629 tablet_copy_service.cc:140] P b4bfd04aeec54be189cc56e88fce86a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1a1135f0b36f4ef2a7b100e4de7a9b8f ({username='oryx', principal='oryx/127.29.71.189@KRBTEST.COM'} at 127.0.0.1:41784)
I20260929 18:53:04.928061 3629 tablet_copy_service.cc:161] P b4bfd04aeec54be189cc56e88fce86a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1a1135f0b36f4ef2a7b100e4de7a9b8f at {username='oryx', principal='oryx/127.29.71.189@KRBTEST.COM'} at 127.0.0.1:41784: session id = 1a1135f0b36f4ef2a7b100e4de7a9b8f-00000000000000000000000000000000
I20260929 18:53:04.928891 3629 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:53:04.929597 4073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:53:04.930888 4073 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: tablet copy: Starting download of 0 data blocks...
I20260929 18:53:04.931046 4073 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: tablet copy: Starting download of 1 WAL segments...
I20260929 18:53:04.931967 4073 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:53:04.932852 3629 tablet_copy_service.cc:342] P b4bfd04aeec54be189cc56e88fce86a2: Request end of tablet copy session 1a1135f0b36f4ef2a7b100e4de7a9b8f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.71.189@KRBTEST.COM'} at 127.0.0.1:41784
I20260929 18:53:04.932911 3629 tablet_copy_service.cc:433] P b4bfd04aeec54be189cc56e88fce86a2: ending tablet copy session 1a1135f0b36f4ef2a7b100e4de7a9b8f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1a1135f0b36f4ef2a7b100e4de7a9b8f
I20260929 18:53:04.935552 4073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:04.936712 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.936748 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:04.936949 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:04.936960 4073 server_base.cc:1034] running on GCE node
I20260929 18:53:04.937142 4073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:04.937309 4073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:04.938426 4073 hybrid_clock.cc:648] HybridClock initialized: now 1790707984938416 us; error 23 us; skew 500 ppm
Sep 29 18:53:04 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:04.939919 4073 init.cc:377] Logged in from keytab as oryx/127.29.71.189@KRBTEST.COM (short username oryx)
I20260929 18:53:04.940241 4073 webserver.cc:460] Webserver started at http://127.29.71.189:42287/ using document root <none> and password file <none>
I20260929 18:53:04.940398 4073 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:04.940447 4073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:04.941075 4073 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:04.941601 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.941787 4073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:04.941850 4073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "e39a7af23417df8b6aed19e6c662d1c5"
server_key_iv: "871486dff4917d5ab9b5bc1225efcf6d"
server_key_version: "encryptionkey@0"
I20260929 18:53:04.941901 4073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:04.949618 4073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:04.951399 4073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:04.954339 4073 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.189:43121
I20260929 18:53:04.954717 4248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.189:43121 every 8 connection(s)
I20260929 18:53:04.954799 4073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
I20260929 18:53:04.955799 4249 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:53:04.956272 4249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Bootstrap starting.
I20260929 18:53:04.958012 4249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:53:04.958282 4249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: Bootstrap complete.
I20260929 18:53:04.958490 4249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } }
I20260929 18:53:04.958559 4249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1a1135f0b36f4ef2a7b100e4de7a9b8f, State: Initialized, Role: LEARNER
I20260929 18:53:04.958586 4249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } }
I20260929 18:53:04.958791 4249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:04.958773 4252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:04.958869 4252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: LEARNER
I20260929 18:53:04.960222 4263 catalog_manager.cc:1259] Loaded cluster ID: f8e56dbb12c3427aa58df443e60df5d2
I20260929 18:53:04.960291 4263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: loading cluster ID for follower catalog manager: success
I20260929 18:53:04.961197 4263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: acquiring CA information for follower catalog manager: success
I20260929 18:53:04.961475 4263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:53:04.968745 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:04.962385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.29.71.189:43121)
0929 18:53:04.962491 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:04.962494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:04.962502 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:04.962524 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:04.962525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:04.962534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:04.962622 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:04.963202 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.963841 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.964562 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:04.964805 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:04.964901 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:04.966109 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:04.966110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:04.966111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:04.966117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:04.967588 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.967953 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.967954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.967955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.968016 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:04.968210 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:04.968211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:04.968211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:04.968323 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:04.968426 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:04.968541 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:53:04.968617 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.29.71.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.188.keytab.
Entry for principal oryx/127.29.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.188.keytab.
I20260929 18:53:05.002147 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:38117
--webserver_interface=127.29.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.188.keytab
--principal=oryx/127.29.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:05.091861 4267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:05.092077 4267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:05.092097 4267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:05.093164 4267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:05.093195 4267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:05.093209 4267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:05.093221 4267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:05.093233 4267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:05.094700 4267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.188.keytab
--principal=oryx/127.29.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:05.094898 4267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:05.095067 4267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:05.099979 4273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:05.100171 4267 server_base.cc:1034] running on GCE node
W20260929 18:53:05.100005 4272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.100004 4275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:05.100528 4267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:05.100817 4267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:05.101984 4267 hybrid_clock.cc:648] HybridClock initialized: now 1790707985101989 us; error 56 us; skew 500 ppm
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:05.104713 4267 init.cc:377] Logged in from keytab as oryx/127.29.71.188@KRBTEST.COM (short username oryx)
I20260929 18:53:05.105233 4267 webserver.cc:460] Webserver started at http://127.29.71.188:45465/ using document root <none> and password file <none>
I20260929 18:53:05.105474 4267 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:05.105520 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:05.105671 4267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:05.106829 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/instance:
uuid: "b3ed090adf8c423bab5446613b820926"
format_stamp: "Formatted at 2026-09-29 18:53:05 on dist-test-slave-q2kh"
server_key: "38c70a7532f7eeab7d33b4e3fd0e1920"
server_key_iv: "5c6d5149786d3a9f75cda2590b29f2e3"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.107208 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal/instance:
uuid: "b3ed090adf8c423bab5446613b820926"
format_stamp: "Formatted at 2026-09-29 18:53:05 on dist-test-slave-q2kh"
server_key: "38c70a7532f7eeab7d33b4e3fd0e1920"
server_key_iv: "5c6d5149786d3a9f75cda2590b29f2e3"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.108575 4267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:53:05.109344 4282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.109582 4267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:05.109673 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
uuid: "b3ed090adf8c423bab5446613b820926"
format_stamp: "Formatted at 2026-09-29 18:53:05 on dist-test-slave-q2kh"
server_key: "38c70a7532f7eeab7d33b4e3fd0e1920"
server_key_iv: "5c6d5149786d3a9f75cda2590b29f2e3"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.109755 4267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:05.118629 4267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:05.121241 4267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:05.121428 4267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:05.125238 4267 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.188:38117
I20260929 18:53:05.125298 4334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.188:38117 every 8 connection(s)
I20260929 18:53:05.125643 4267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
I20260929 18:53:05.126748 4335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:05.127626 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 4267
I20260929 18:53:05.128551 4335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.188@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.71.188@KRBTEST.COM
I20260929 18:53:05.139375 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.129357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.29.71.190:33865)
0929 18:53:05.129407 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.129409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.129485 (+ 76us) server_negotiation.cc:407] Connection header received
0929 18:53:05.129740 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.129741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.129753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.129817 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.130679 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.131526 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.132876 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.133199 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.133275 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.135766 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.135768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.135768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.135777 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.137441 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.137972 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.137973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.137974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.138028 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.138623 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.138624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.138625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.138787 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.138859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.139165 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.139250 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:53:05.139513 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.129209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:41790)
0929 18:53:05.129373 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.129410 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.129626 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.129880 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.129883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.130332 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.130588 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.130595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.131640 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.131641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.132702 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.132704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.133303 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.133304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.133400 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.133973 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.133983 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.135673 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.137523 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.137528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.137530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.137890 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.138428 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.138429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.138429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.138550 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.138867 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.138871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.139016 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.139101 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.139262 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":18,"thread_start_us":96,"threads_started":1}
I20260929 18:53:05.139513 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.128361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.29.71.188:38117)
0929 18:53:05.128955 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.128965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.128998 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:53:05.129596 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.129604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.129626 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.129785 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.130618 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.131786 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.132875 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.133392 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.133455 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.136303 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.136310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.136311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.136318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.138032 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.138417 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.138420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.138421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.138526 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.138746 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.138746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.138747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.139025 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.139156 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.139270 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.139328 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":548,"thread_start_us":488,"threads_started":1}
I20260929 18:53:05.140260 4335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.188@KRBTEST.COM for oryx/127.29.71.189@KRBTEST.COM
I20260929 18:53:05.146795 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.140518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.0.0.1:38246)
0929 18:53:05.140557 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.140568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.140609 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.140740 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.140740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.140892 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.140929 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.140931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.141600 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.141601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.142246 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.142248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.142640 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.142640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.142705 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.143144 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.143148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.144530 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.145933 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.145933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.145934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.146088 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.146338 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.146338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.146338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.146386 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.146575 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.146576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.146610 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.146660 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.146728 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 18:53:05.146799 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.140592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.29.71.189:43121)
0929 18:53:05.140632 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.140634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.140643 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:05.140670 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.140671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.140683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.140729 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.141013 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.141506 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.142343 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.142560 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.142641 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.144596 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.144598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.144598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.144605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.145837 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.146146 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.146147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.146147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.146257 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.146427 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.146427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.146427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.146523 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.146571 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.146676 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.146739 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:53:05.147326 4335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 38117 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.188@KRBTEST.COM for oryx/127.29.71.188@KRBTEST.COM
I20260929 18:53:05.154508 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.147631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.0.0.1:34502)
0929 18:53:05.147674 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.147685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.147734 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.147881 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.147882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.147990 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.148021 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.148022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.148651 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.148651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.149152 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.149154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.149484 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.149485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.149520 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.149919 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.149931 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.151221 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.153246 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.153247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.153247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.153491 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.153717 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.153717 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.153718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.153792 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.154085 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.154086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.154126 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.154229 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.154356 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 18:53:05.154610 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.147658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.29.71.188:38117)
0929 18:53:05.147728 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.147730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.147739 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:05.147802 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.147803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.147810 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.147847 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.148079 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.148588 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.149220 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.149422 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.149475 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.151324 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.151326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.151326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.151333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.153147 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.153585 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.153586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.153587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.153645 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.153861 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.153862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.153862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.154012 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.154086 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.154257 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.154442 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":51}
I20260929 18:53:05.156514 4335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Bootstrap starting.
I20260929 18:53:05.157507 4335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:05.157825 4335 log.cc:824] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:05.158702 4335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: No bootstrap required, opened a new log
I20260929 18:53:05.160359 4335 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 38117 } }
I20260929 18:53:05.160488 4335 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:05.160511 4335 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ed090adf8c423bab5446613b820926, State: Initialized, Role: FOLLOWER
I20260929 18:53:05.160631 4335 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 38117 } }
I20260929 18:53:05.160902 4335 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:05.160923 4340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 38117 } } }
I20260929 18:53:05.161019 4340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:53:05.162631 4351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:53:05.162724 4351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:53:05.164883 4267 master_runner.cc:428] Detected that this master b3ed090adf8c423bab5446613b820926 is joining an existing cluster
I20260929 18:53:05.164937 4267 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.71.188:38117
I20260929 18:53:05.174830 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.166318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:41800)
0929 18:53:05.166491 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.166503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.166588 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.166783 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.166784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.167132 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.167240 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.167242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.168605 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.168606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.169554 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.169557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.170702 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.170703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.170773 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.171579 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.171583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.171894 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.173831 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.173832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.173832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.174002 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.174248 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.174249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.174249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.174312 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.174581 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.174581 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.174616 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.174679 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.174754 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":49,"thread_start_us":87,"threads_started":1}
I20260929 18:53:05.174893 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.166208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.29.71.190:33865)
0929 18:53:05.166249 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.166251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.166561 (+ 310us) server_negotiation.cc:407] Connection header received
0929 18:53:05.166667 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.166668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.166680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.166715 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.167741 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.168505 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.169654 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.170578 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.170625 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.171990 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.171992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.171992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.172000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.173743 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.174076 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.174077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.174077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.174138 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.174380 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.174381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.174381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.174514 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.174585 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.174719 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.174796 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707984, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.190@KRBTEST.COM for oryx/127.29.71.188@KRBTEST.COM
I20260929 18:53:05.189911 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.166320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.0.0.1:38250)
0929 18:53:05.166491 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.166503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.166588 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.166785 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.166786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.167128 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.167689 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.167690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.168640 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.168640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.169557 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.169559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.170057 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.170057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.170130 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.170569 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.170575 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.186605 (+ 16030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.188528 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.188529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.188529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.188951 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.189216 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.189218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.189283 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.189576 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.189577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.189621 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.189682 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.189790 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":41,"thread_start_us":86,"threads_started":1}
I20260929 18:53:05.190120 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.166458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.0.0.1:34512)
0929 18:53:05.166686 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.166695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.166763 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.166913 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.166913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.167225 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.167285 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.167287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.168180 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.168181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.169042 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.169045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.169539 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.169539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.169591 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.170062 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.170067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.186834 (+ 16767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.188955 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.188956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.188956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.189285 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.189483 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.189483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.189532 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.189792 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.189792 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.189830 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.189885 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.189984 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
I20260929 18:53:05.190284 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.166611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.29.71.188:38117)
0929 18:53:05.166646 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.166648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.166744 (+ 96us) server_negotiation.cc:407] Connection header received
0929 18:53:05.166827 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.166828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.166836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.166863 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.167367 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.168092 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.169130 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.169411 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.169458 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.186922 (+ 17464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.186923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.186924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.186935 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.188873 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189365 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.189366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.189367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.189417 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189597 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.189598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.189598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.189720 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.190079 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.190117 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.190209 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:53:05.190740 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.166192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.29.71.189:43121)
0929 18:53:05.166238 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.166241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.166561 (+ 320us) server_negotiation.cc:407] Connection header received
0929 18:53:05.166676 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.166677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.166687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.166717 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.167778 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.168544 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.169654 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.169937 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.169978 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.186711 (+ 16733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.186713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.186714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.186722 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.188438 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189054 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.189056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.189057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.189137 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189348 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.189349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.189349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.189492 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.189551 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.190564 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.190649 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:53:05.190971 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3671
I20260929 18:53:05.191641 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.177393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.0.0.1:34528)
0929 18:53:05.177453 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.177469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.177509 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.177913 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.177914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.178065 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.178104 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.178106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.178844 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.178845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.179366 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.179368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.179784 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.179785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.179858 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.180266 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.180270 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.181723 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.189378 (+ 7655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.189380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.189637 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.189848 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.189850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.189905 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.191276 (+ 1371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.191278 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.191326 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.191375 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.191492 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 18:53:05.192229 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.177573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.29.71.188:38117)
0929 18:53:05.177717 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.177719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.177727 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:05.177770 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.177771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.177822 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.177908 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.178234 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.178731 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.179466 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.179690 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.179789 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.181876 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.181877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.181878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.181884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.189282 (+ 7398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.189714 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.189715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.189715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.189760 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.191014 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.191016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.191016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.191190 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.192037 (+ 847us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.192078 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.192161 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260929 18:53:05.192688 3609 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.71.188:38117: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b3ed090adf8c423bab5446613b820926"
member_type: NON_VOTER
last_known_addr {
host: "127.29.71.188"
port: 38117
}
attrs {
promote: true
}
}
dest_uuid: "b4bfd04aeec54be189cc56e88fce86a2"
cas_config_opid_index: 5
I20260929 18:53:05.192974 3609 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:05.193473 4169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3ed090adf8c423bab5446613b820926 (127.29.71.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.194401 4169 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.71.188:38117
I20260929 18:53:05.194478 4169 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.194556 4169 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:05.195163 4309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:05.195887 4309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 FOLLOWER]: Refusing update from remote peer b4bfd04aeec54be189cc56e88fce86a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:53:05.196166 4168 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:05.197388 4309 raft_consensus.cc:858] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 38117 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
W20260929 18:53:05.197445 4309 raft_consensus.cc:1565] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 38117 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:05.197790 4267 master.cc:537] Master@127.29.71.188:38117 shutting down...
I20260929 18:53:05.198118 4309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1a1135f0b36f4ef2a7b100e4de7a9b8f (127.29.71.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:05.200529 4368 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:05.200619 4368 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:53:05.200659 4368 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:05.200704 4368 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:53:05.200748 4340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bfd04aeec54be189cc56e88fce86a2. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } }
I20260929 18:53:05.200793 4340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:53:05.201136 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.201236 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:05.201344 4362 mvcc.cc:205] Tried to move back new op lower bound from 7334739903053168640 to 7334739902944542720. Current Snapshot: MvccSnapshot[applied={T|T < 7334739902955274240}]
W20260929 18:53:05.201678 3594 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 18:53:05.201864 4267 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 18:53:05.201937 4267 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 18:53:05.201969 4267 tablet_replica.cc:330] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: stopping tablet replica
W20260929 18:53:05.202142 3594 consensus_peers.cc:487] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 -> Peer b3ed090adf8c423bab5446613b820926 (127.29.71.188:38117): Couldn't send request to peer b3ed090adf8c423bab5446613b820926. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:53:05.203315 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.194781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.0.0.1:38256)
0929 18:53:05.196069 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.196081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.196118 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.199756 (+ 3638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.199758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.199760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:05.199810 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.199812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.200735 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.200736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.202240 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.202243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.203030 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.203031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.203110 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.203132 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.203199 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.203246 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268}
I20260929 18:53:05.203663 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:34961
--local_ip_for_outbound_sockets=127.29.71.129
--tserver_master_addrs=127.29.71.190:33865
--webserver_port=45623
--webserver_interface=127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab
--principal=oryx/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:53:05.203959 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.194991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.29.71.189:43121)
0929 18:53:05.199557 (+ 4566us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.199559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.199569 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:05.199592 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.199593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.199616 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.199684 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:05.199881 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.200638 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.202332 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.202942 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.203017 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.203084 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.203626 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.203670 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4541}
I20260929 18:53:05.205286 4223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Refusing update from remote peer b4bfd04aeec54be189cc56e88fce86a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:53:05.205542 4361 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 18:53:05.205974 4223 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3ed090adf8c423bab5446613b820926 (127.29.71.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.207340 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.207377 4252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bfd04aeec54be189cc56e88fce86a2. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: NON_VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: true } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.207445 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: LEARNER
I20260929 18:53:05.207466 4252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: LEARNER
I20260929 18:53:05.207517 4361 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: attempting to promote NON_VOTER 1a1135f0b36f4ef2a7b100e4de7a9b8f to VOTER
I20260929 18:53:05.207705 4361 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:05.208168 4223 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 LEARNER]: Refusing update from remote peer b4bfd04aeec54be189cc56e88fce86a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 18:53:05.208202 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.208090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.0.0.1:34530)
0929 18:53:05.208135 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.208173 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:53:05.208462 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.208395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.0.0.1:34532)
0929 18:53:05.208426 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.208444 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:38117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 18:53:05.208482 4169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 18:53:05.208557 3594 consensus_peers.cc:487] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 -> Peer b3ed090adf8c423bab5446613b820926 (127.29.71.188:38117): Couldn't send request to peer b3ed090adf8c423bab5446613b820926. Status: Network error: Client connection negotiation failed: client connection to 127.29.71.188:38117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:53:05.209133 4168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1a1135f0b36f4ef2a7b100e4de7a9b8f (127.29.71.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.209357 4223 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1a1135f0b36f4ef2a7b100e4de7a9b8f (127.29.71.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.210038 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.210119 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:05.210193 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.210235 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:05.210563 4252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bfd04aeec54be189cc56e88fce86a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.210669 4252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:53:05.218353 4267 master.cc:552] Master@127.29.71.188:38117 shutdown complete.
I20260929 18:53:05.218410 4267 master_runner.cc:305] Clearing existing system tablet
I20260929 18:53:05.218557 4267 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:05.218602 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:05.219348 4267 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:05.219894 4374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.220096 4267 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:05.220153 4267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:05.220177 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
uuid: "b3ed090adf8c423bab5446613b820926"
format_stamp: "Formatted at 2026-09-29 18:53:05 on dist-test-slave-q2kh"
server_key: "38c70a7532f7eeab7d33b4e3fd0e1920"
server_key_iv: "5c6d5149786d3a9f75cda2590b29f2e3"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.220920 4267 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:05.222369 4267 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:53:05.222436 4267 log.cc:1196] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 18:53:05.222666 4267 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Deleting consensus metadata
I20260929 18:53:05.222781 4267 master_runner.cc:315] Copying system tablet from 127.29.71.190:33865
I20260929 18:53:05.223381 4267 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: tablet copy: Beginning tablet copy session from remote peer at address 127.29.71.190:33865
I20260929 18:53:05.230768 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.223684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:41804)
0929 18:53:05.223821 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.223834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.223909 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.224045 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.224046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.224241 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.224322 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.224323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.225272 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.225273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.226194 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.226196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.226673 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.226674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.226743 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.227259 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.227264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.227667 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.229525 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.229526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.229527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.229737 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.230033 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.230034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.230034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.230096 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.230417 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.230418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.230465 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.230549 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.230651 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260929 18:53:05.230782 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.223845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.29.71.190:33865)
0929 18:53:05.223887 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.223889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.223902 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:53:05.223969 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.223970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.223980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.224041 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.224406 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.225188 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.226272 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.226596 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.226669 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.227763 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.227764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.227765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.227772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.229438 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.229873 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.229874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.229875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.229948 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.230184 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.230185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.230186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.230342 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.230423 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.230596 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.230684 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:53:05.231419 3629 tablet_copy_service.cc:140] P b4bfd04aeec54be189cc56e88fce86a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b3ed090adf8c423bab5446613b820926 ({username='oryx', principal='oryx/127.29.71.188@KRBTEST.COM'} at 127.0.0.1:41804)
I20260929 18:53:05.231485 3629 tablet_copy_service.cc:161] P b4bfd04aeec54be189cc56e88fce86a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b3ed090adf8c423bab5446613b820926 at {username='oryx', principal='oryx/127.29.71.188@KRBTEST.COM'} at 127.0.0.1:41804: session id = b3ed090adf8c423bab5446613b820926-00000000000000000000000000000000
I20260929 18:53:05.232033 3629 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:53:05.233469 4267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:53:05.234714 4267 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: tablet copy: Starting download of 0 data blocks...
I20260929 18:53:05.234931 4267 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: tablet copy: Starting download of 1 WAL segments...
I20260929 18:53:05.235693 4267 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:53:05.236665 3629 tablet_copy_service.cc:342] P b4bfd04aeec54be189cc56e88fce86a2: Request end of tablet copy session b3ed090adf8c423bab5446613b820926-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.71.188@KRBTEST.COM'} at 127.0.0.1:41804
I20260929 18:53:05.236723 3629 tablet_copy_service.cc:433] P b4bfd04aeec54be189cc56e88fce86a2: ending tablet copy session b3ed090adf8c423bab5446613b820926-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b3ed090adf8c423bab5446613b820926
I20260929 18:53:05.239495 4267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:05.240758 4381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.240792 4382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.243180 4384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:05.243278 4267 server_base.cc:1034] running on GCE node
I20260929 18:53:05.243466 4267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:05.243613 4267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:05.244740 4267 hybrid_clock.cc:648] HybridClock initialized: now 1790707985244730 us; error 26 us; skew 500 ppm
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:05.246206 4267 init.cc:377] Logged in from keytab as oryx/127.29.71.188@KRBTEST.COM (short username oryx)
I20260929 18:53:05.246440 4267 webserver.cc:460] Webserver started at http://127.29.71.188:37223/ using document root <none> and password file <none>
I20260929 18:53:05.246556 4267 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:05.246583 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:05.247318 4267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:05.247936 4391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.248126 4267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:05.248178 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
uuid: "b3ed090adf8c423bab5446613b820926"
format_stamp: "Formatted at 2026-09-29 18:53:05 on dist-test-slave-q2kh"
server_key: "38c70a7532f7eeab7d33b4e3fd0e1920"
server_key_iv: "5c6d5149786d3a9f75cda2590b29f2e3"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.248234 4267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:05.261587 4267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:05.263600 4267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:05.266824 4267 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.188:38117
I20260929 18:53:05.267269 4267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
I20260929 18:53:05.267570 4443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.188:38117 every 8 connection(s)
I20260929 18:53:05.268326 4444 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:53:05.268800 4444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Bootstrap starting.
I20260929 18:53:05.271144 4444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:53:05.271346 4444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: Bootstrap complete.
I20260929 18:53:05.271545 4444 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:05.271629 4444 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3ed090adf8c423bab5446613b820926, State: Initialized, Role: LEARNER
I20260929 18:53:05.271662 4444 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:05.271816 4444 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:05.273094 4447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } } }
I20260929 18:53:05.273202 4447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:53:05.273515 4458 catalog_manager.cc:1259] Loaded cluster ID: f8e56dbb12c3427aa58df443e60df5d2
I20260929 18:53:05.273612 4458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: loading cluster ID for follower catalog manager: success
I20260929 18:53:05.274797 4458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: acquiring CA information for follower catalog manager: success
I20260929 18:53:05.275170 4458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3ed090adf8c423bab5446613b820926: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:53:05.307440 4370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:05.307648 4370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:05.307672 4370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:05.308811 4370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:05.308856 4370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:05.308882 4370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:05.310417 4370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab
--principal=oryx/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:34961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=45623
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:05.310652 4370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:05.310892 4370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:05.315856 4466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.315907 4464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.315883 4463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:05.316470 4370 server_base.cc:1034] running on GCE node
I20260929 18:53:05.316692 4370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:05.316942 4370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:05.318116 4370 hybrid_clock.cc:648] HybridClock initialized: now 1790707985318103 us; error 35 us; skew 500 ppm
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:05.320609 4370 init.cc:377] Logged in from keytab as oryx/127.29.71.129@KRBTEST.COM (short username oryx)
I20260929 18:53:05.321087 4370 webserver.cc:460] Webserver started at http://127.29.71.129:45623/ using document root <none> and password file <none>
I20260929 18:53:05.321331 4370 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:05.321416 4370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:05.322811 4370 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:53:05.323405 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.323590 4370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:05.323695 4370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "0d1f56ca3d184435b0e9a08383d25abf"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "95082266dc164a1bd1deba53a4c8d3be"
server_key_iv: "3f16a65909cadcefca80d7573a88e961"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.324025 4370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:05.341820 4370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:05.344262 4370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:05.344412 4370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:05.344657 4370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:05.344986 4370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:05.345026 4370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.345067 4370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:05.345093 4370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.351835 4370 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:34961
I20260929 18:53:05.351945 4586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:34961 every 8 connection(s)
I20260929 18:53:05.352257 4370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.189@KRBTEST.COM
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.188@KRBTEST.COM
I20260929 18:53:05.361807 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 4370
I20260929 18:53:05.362028 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3805
I20260929 18:53:05.364043 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.353106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:32779 (local address 127.29.71.189:43121)
0929 18:53:05.353155 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.353157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.353395 (+ 238us) server_negotiation.cc:407] Connection header received
0929 18:53:05.354431 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.354432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.354442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.354513 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.355495 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.356288 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357451 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.357736 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357803 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.360497 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.360499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.360499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.360508 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.362354 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.362861 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.362862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.362862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.362918 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363193 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.363194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.363194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.363378 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.363434 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.363819 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.363905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:53:05.364665 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.352992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.29.71.129:32779)
0929 18:53:05.353238 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.353283 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.354335 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.354609 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.354618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.355095 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.355381 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.355388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.356376 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.356377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.357335 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.357337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.357828 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.357981 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.358756 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.358761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.360405 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.362451 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.362457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.362459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.362773 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.362990 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.362991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.362991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.363122 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.363482 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.363488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.363660 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.364179 (+ 519us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.364368 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":149,"thread_start_us":86,"threads_started":1}
I20260929 18:53:05.365216 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.352992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.29.71.129:50217)
0929 18:53:05.354655 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.354664 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.354704 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.354888 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.354889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.355089 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.355380 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.355385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.356261 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.356262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.357477 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.357480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.357891 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.357983 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.358590 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.358601 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.361457 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.363401 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.363402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.363696 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.363904 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.363905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.363974 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.364569 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.364571 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.364622 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.364686 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.365107 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"mutex_wait_us":195,"thread_start_us":74,"threads_started":1}
I20260929 18:53:05.365300 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.353321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:39719 (local address 127.29.71.190:33865)
0929 18:53:05.353401 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.353404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.354555 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 18:53:05.354612 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.354614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.354627 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.354689 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.355495 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.356259 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357557 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.357863 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357923 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.360834 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.360836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.360836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.360845 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.362951 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363348 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.363349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.363350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.363409 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363614 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.363615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.363615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.364880 (+ 1265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.364927 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.365124 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.365197 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:53:05.365592 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.352991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:39719)
0929 18:53:05.354483 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.354491 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.354558 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.354703 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.354703 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.355089 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.355382 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.355388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.356328 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.356329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.357487 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.357489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.357974 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.358026 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.358597 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.358601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.360741 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.363047 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.363048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.363263 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.363482 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.363482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.363540 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.364973 (+ 1433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.364974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.365013 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.365409 (+ 396us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.365499 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"mutex_wait_us":169,"thread_start_us":102,"threads_started":1}
I20260929 18:53:05.365587 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.353299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:50217 (local address 127.29.71.188:38117)
0929 18:53:05.353411 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.353413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.354752 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 18:53:05.354784 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.354785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.354795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.354867 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.355490 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.356157 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357576 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.357818 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.357880 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.361570 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.361572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.361572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.361579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.363310 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.363772 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.363774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.363774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.363831 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.364046 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.364047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.364048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.364461 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.364538 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.365405 (+ 867us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.365493 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260929 18:53:05.365944 4588 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33865
I20260929 18:53:05.366034 4588 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.366225 4588 heartbeater.cc:511] Master 127.29.71.190:33865 requested a full tablet report, sending...
I20260929 18:53:05.366519 4587 heartbeater.cc:347] Connected to a master server at 127.29.71.188:38117
I20260929 18:53:05.366585 4587 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.366647 3609 ts_manager.cc:194] Re-registered known tserver with Master: 0d1f56ca3d184435b0e9a08383d25abf (127.29.71.129:34961)
I20260929 18:53:05.366657 4587 heartbeater.cc:511] Master 127.29.71.188:38117 requested a full tablet report, sending...
I20260929 18:53:05.367038 4399 ts_manager.cc:194] Registered new tserver with Master: 0d1f56ca3d184435b0e9a08383d25abf (127.29.71.129:34961)
I20260929 18:53:05.367107 3609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:39719
I20260929 18:53:05.369792 4589 heartbeater.cc:347] Connected to a master server at 127.29.71.189:43121
I20260929 18:53:05.369863 4589 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.369935 4589 heartbeater.cc:511] Master 127.29.71.189:43121 requested a full tablet report, sending...
I20260929 18:53:05.370091 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:33565
--local_ip_for_outbound_sockets=127.29.71.130
--tserver_master_addrs=127.29.71.190:33865
--webserver_port=45723
--webserver_interface=127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab
--principal=oryx/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:53:05.370401 4213 ts_manager.cc:194] Registered new tserver with Master: 0d1f56ca3d184435b0e9a08383d25abf (127.29.71.129:34961)
W20260929 18:53:05.453893 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:05.454140 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:05.454190 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:05.455328 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:05.455391 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:05.455431 4595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:05.456887 4595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab
--principal=oryx/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:33565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=45723
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:05.457155 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:05.457388 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:53:05.462633 4595 server_base.cc:1034] running on GCE node
W20260929 18:53:05.462610 4600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.462610 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.462617 4601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:05.463009 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:05.463248 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:05.464427 4595 hybrid_clock.cc:648] HybridClock initialized: now 1790707985464394 us; error 29 us; skew 500 ppm
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:05.467046 4595 init.cc:377] Logged in from keytab as oryx/127.29.71.130@KRBTEST.COM (short username oryx)
I20260929 18:53:05.467468 4595 webserver.cc:460] Webserver started at http://127.29.71.130:45723/ using document root <none> and password file <none>
I20260929 18:53:05.467702 4595 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:05.467759 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:05.469097 4595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:05.469657 4610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.469838 4595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:05.469935 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "ed16a57445e028e438b7f8a044f078cb"
server_key_iv: "a720aa73448177f736833b4d2b6dc4d6"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.470319 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:05.509658 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:05.511976 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:05.512157 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:05.512428 4595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:05.512822 4595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:05.512878 4595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.512919 4595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:05.512945 4595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.519652 4595 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:33565
I20260929 18:53:05.519691 4723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:33565 every 8 connection(s)
I20260929 18:53:05.520082 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:05.525537 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 4595
I20260929 18:53:05.525729 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3939
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.189@KRBTEST.COM
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.188@KRBTEST.COM
I20260929 18:53:05.531888 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.521106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:43525 (local address 127.29.71.188:38117)
0929 18:53:05.521148 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.521150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.521159 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:05.521755 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.521756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.521769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.521802 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.523008 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.523762 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.524908 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.525245 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.525313 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.529163 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.529165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.529165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.529173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.530689 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.530997 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.530998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.530999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.531055 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.531260 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.531261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.531261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.531391 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.531449 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.531696 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.531782 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:53:05.531955 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.520813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:58497 (local address 127.29.71.189:43121)
0929 18:53:05.520860 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.520863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.521138 (+ 275us) server_negotiation.cc:407] Connection header received
0929 18:53:05.521731 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.521733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.521745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.521800 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.523007 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.523758 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.524898 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.525199 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.525245 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.528332 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.528333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.528333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.528341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.530104 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.530620 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.530621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.530621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.530685 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.531197 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.531198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.531199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.531340 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.531411 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.531712 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.531799 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:53:05.532401 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.520818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.29.71.130:58497)
0929 18:53:05.520983 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.521025 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.521634 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.522215 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.522216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.522670 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.522892 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.522898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.523858 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.523858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.524786 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.524789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.525343 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.525343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.525413 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.526321 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.526326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.528209 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.530222 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.530229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.530231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.530538 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.531016 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.531017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.531017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.531144 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.531409 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.531415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.531571 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.532039 (+ 468us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.532157 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":66,"thread_start_us":87,"threads_started":1}
I20260929 18:53:05.532590 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.520818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.29.71.130:43525)
0929 18:53:05.521023 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.521035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.521636 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.521890 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.521895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.522694 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.522892 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.522898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.523858 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.523858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.524799 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.524801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.526380 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.526381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.526457 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.526847 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.526852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.529087 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.530764 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.530765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.530765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.530924 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.531126 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.531127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.531128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.531199 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.531474 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.531475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.531564 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.531677 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.531850 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":37,"thread_start_us":117,"threads_started":1}
I20260929 18:53:05.532860 4726 heartbeater.cc:347] Connected to a master server at 127.29.71.189:43121
I20260929 18:53:05.532884 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:44429
--local_ip_for_outbound_sockets=127.29.71.131
--tserver_master_addrs=127.29.71.190:33865
--webserver_port=37105
--webserver_interface=127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab
--principal=oryx/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:53:05.532974 4726 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.533072 4724 heartbeater.cc:347] Connected to a master server at 127.29.71.188:38117
I20260929 18:53:05.533120 4724 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.533169 4726 heartbeater.cc:511] Master 127.29.71.189:43121 requested a full tablet report, sending...
I20260929 18:53:05.533200 4724 heartbeater.cc:511] Master 127.29.71.188:38117 requested a full tablet report, sending...
I20260929 18:53:05.533604 4399 ts_manager.cc:194] Registered new tserver with Master: 27a0da3e66f34b39a0a7a0fbbf0bd8c5 (127.29.71.130:33565)
I20260929 18:53:05.533648 4213 ts_manager.cc:194] Registered new tserver with Master: 27a0da3e66f34b39a0a7a0fbbf0bd8c5 (127.29.71.130:33565)
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:05.538120 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.520914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:46519 (local address 127.29.71.190:33865)
0929 18:53:05.520965 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.520967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.521293 (+ 326us) server_negotiation.cc:407] Connection header received
0929 18:53:05.521755 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.521757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.521769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.521802 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.522992 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.523494 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.524554 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.524751 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.524852 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.535465 (+ 10613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.535467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.535467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.535474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.536857 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.537293 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.537294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.537294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.537372 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.537626 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.537627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.537628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.537745 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.537809 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.537993 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.538060 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:53:05.538241 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.520818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:46519)
0929 18:53:05.521230 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.521239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.521639 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.521895 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.521895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.522668 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.522891 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.522897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.523599 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.523600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.524459 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.524461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.524866 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.524867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.524996 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.525530 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.525539 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.535357 (+ 9818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.536941 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.536942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.536943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.537194 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.537452 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.537453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.537454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.537524 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.537818 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.537819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.537879 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.537960 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.538080 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":272,"thread_start_us":58,"threads_started":1}
I20260929 18:53:05.538604 4725 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33865
I20260929 18:53:05.538666 4725 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.538741 4725 heartbeater.cc:511] Master 127.29.71.190:33865 requested a full tablet report, sending...
I20260929 18:53:05.539027 3609 ts_manager.cc:194] Re-registered known tserver with Master: 27a0da3e66f34b39a0a7a0fbbf0bd8c5 (127.29.71.130:33565)
I20260929 18:53:05.539417 3609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:46519
W20260929 18:53:05.615693 4731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:05.615886 4731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:05.615911 4731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:05.617071 4731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:05.617110 4731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:05.617136 4731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:05.618815 4731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab
--principal=oryx/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:44429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=37105
--tserver_master_addrs=127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:05.619061 4731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:05.619311 4731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:05.624253 4736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:05.624289 4737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:05.624447 4731 server_base.cc:1034] running on GCE node
W20260929 18:53:05.624487 4739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:05.624723 4731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:05.624980 4731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:05.626134 4731 hybrid_clock.cc:648] HybridClock initialized: now 1790707985626135 us; error 40 us; skew 500 ppm
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:05.629267 4731 init.cc:377] Logged in from keytab as oryx/127.29.71.131@KRBTEST.COM (short username oryx)
I20260929 18:53:05.629802 4731 webserver.cc:460] Webserver started at http://127.29.71.131:37105/ using document root <none> and password file <none>
I20260929 18:53:05.630069 4731 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:05.630146 4731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:05.631598 4731 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:05.632146 4746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.632318 4731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:05.632419 4731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "52270bceeff84e628caf71897b7e242c"
format_stamp: "Formatted at 2026-09-29 18:53:04 on dist-test-slave-q2kh"
server_key: "dedaf90ff2d9875bf6a2ac363cec9e7d"
server_key_iv: "9927be6b5cf0f9283fa5c5c4c3b5ba54"
server_key_version: "encryptionkey@0"
I20260929 18:53:05.632776 4731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:05.660872 4731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:05.663280 4731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:05.663446 4731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:05.663689 4731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:05.664004 4731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:05.664057 4731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.664098 4731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:05.664126 4731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:05.670154 4731 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:44429
I20260929 18:53:05.670229 4859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:44429 every 8 connection(s)
I20260929 18:53:05.670552 4731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.188@KRBTEST.COM
I20260929 18:53:05.678388 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 4731
Sep 29 18:53:05 dist-test-slave-q2kh krb5kdc[3564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707985, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.189@KRBTEST.COM
I20260929 18:53:05.683176 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.671254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:60977 (local address 127.29.71.188:38117)
0929 18:53:05.671298 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.671300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.671599 (+ 299us) server_negotiation.cc:407] Connection header received
0929 18:53:05.672729 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.672731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.672743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.672801 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.673720 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.674358 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675370 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.675605 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675669 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.679235 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.679236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.679237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.679244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.681968 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682372 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.682373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.682373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.682427 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682609 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.682610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.682610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.682722 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.682775 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.683016 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.683082 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:53:05.683176 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.671250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:40545 (local address 127.29.71.190:33865)
0929 18:53:05.671297 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.671299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.671856 (+ 557us) server_negotiation.cc:407] Connection header received
0929 18:53:05.672993 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.672995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.673027 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.673091 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.673736 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.674532 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675647 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.675858 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675944 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.678217 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.678219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.678220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.678227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.680783 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682089 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.682091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.682091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.682145 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682446 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.682447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.682447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.682584 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.682637 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.683001 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.683083 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:53:05.683336 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.671648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:43121 (local address 127.29.71.131:60563)
0929 18:53:05.672807 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.672818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.672866 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.673047 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.673048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.673417 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.673624 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.673630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.674634 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.674635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.675541 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.675542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.676012 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.676013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.676077 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.676753 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.676758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.679389 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.681793 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.681794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.681795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.681999 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.682259 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.682260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.682373 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.682651 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.682657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.682814 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.682913 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.683085 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":73,"thread_start_us":67,"threads_started":1}
I20260929 18:53:05.683605 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.671275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:38117 (local address 127.29.71.131:60977)
0929 18:53:05.671459 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.671503 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.672609 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.672910 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.672915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.673404 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.673622 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.673627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.674459 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.674460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.675256 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.675258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.675728 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.675891 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.676415 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.676423 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.679106 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.682045 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.682047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.682285 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.682495 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.682496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.682545 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.682798 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.682799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.682829 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.682922 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.683526 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":472,"thread_start_us":118,"threads_started":1}
I20260929 18:53:05.683842 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.671276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:40545)
0929 18:53:05.671767 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.671776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.672920 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.673091 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.673092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.673411 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:05.673624 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.673630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.674625 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.674626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.675539 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.675542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.675944 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.676008 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.676559 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:05.676565 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:05.678068 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:05.680895 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.680902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.680904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.681212 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.682239 (+ 1027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:05.682240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:05.682373 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:05.682670 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:05.682671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:05.682814 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.682915 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.683688 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":705,"thread_start_us":121,"threads_started":1}
I20260929 18:53:05.683926 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.671749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:60563 (local address 127.29.71.189:43121)
0929 18:53:05.671802 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.671804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.672863 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 18:53:05.672929 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.672930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.672941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.672990 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.673735 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.674526 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675625 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.675894 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.675958 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.679804 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.679805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.679806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.679813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.681718 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682107 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.682108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.682108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.682191 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.682444 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.682444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.682445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.682570 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.682628 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.683767 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.683839 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:53:05.684299 4860 heartbeater.cc:347] Connected to a master server at 127.29.71.188:38117
I20260929 18:53:05.684386 4860 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.684531 4862 heartbeater.cc:347] Connected to a master server at 127.29.71.189:43121
I20260929 18:53:05.684571 4860 heartbeater.cc:511] Master 127.29.71.188:38117 requested a full tablet report, sending...
I20260929 18:53:05.685027 4399 ts_manager.cc:194] Registered new tserver with Master: 52270bceeff84e628caf71897b7e242c (127.29.71.131:44429)
I20260929 18:53:05.684576 4862 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.685319 4862 heartbeater.cc:511] Master 127.29.71.189:43121 requested a full tablet report, sending...
I20260929 18:53:05.685572 4213 ts_manager.cc:194] Registered new tserver with Master: 52270bceeff84e628caf71897b7e242c (127.29.71.131:44429)
I20260929 18:53:05.684595 4861 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33865
I20260929 18:53:05.685879 4861 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:05.685942 4861 heartbeater.cc:511] Master 127.29.71.190:33865 requested a full tablet report, sending...
I20260929 18:53:05.686196 3609 ts_manager.cc:194] Re-registered known tserver with Master: 52270bceeff84e628caf71897b7e242c (127.29.71.131:44429)
I20260929 18:53:05.686678 3609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:40545
I20260929 18:53:05.690613 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.679850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.29.71.188:38117)
0929 18:53:05.679947 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.679949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.679956 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:53:05.679975 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.679976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.679987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.680836 (+ 849us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.680870 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.681708 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.685168 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.685479 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.685753 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.687279 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.687280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.687281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.687287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.689379 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.689778 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.689779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.689780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.689829 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.690055 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.690056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.690056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.690206 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.690302 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.690441 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.690510 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":56,"threads_started":1}
I20260929 18:53:05.690907 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.679520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.29.71.190:33865)
0929 18:53:05.683256 (+ 3736us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.683258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.683265 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:53:05.683282 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.683283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.683291 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.683378 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.684101 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.684876 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.686351 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.686622 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.686676 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.688090 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.688091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.688092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.688098 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.689223 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.689566 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.689567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.689568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.689629 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.689879 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.689880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.689880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.690004 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.690095 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.690611 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.690802 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3722,"thread_start_us":83,"threads_started":1}
I20260929 18:53:05.691282 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.680186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.29.71.189:43121)
0929 18:53:05.680413 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.680415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.680582 (+ 167us) server_negotiation.cc:407] Connection header received
0929 18:53:05.680615 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.680616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.680626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.680681 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.681347 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.682066 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.683101 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.685339 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.685430 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.687279 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.687280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.687281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.687287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.689656 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.690011 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.690012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.690012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.690065 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.690315 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.690316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.690316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.690544 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.690638 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.691165 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.691223 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":47,"threads_started":1}
I20260929 18:53:05.692391 3609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:53:05.693917 3609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:53:05.701608 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.697354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:44429 (local address 127.0.0.1:59284)
0929 18:53:05.697739 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.697748 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.697791 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.698551 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.698552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.698555 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:05.698600 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.698602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.699936 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.699937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.700697 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.700699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.701366 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.701367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.701423 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.701428 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.701489 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.701532 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":9,"thread_start_us":84,"threads_started":1}
I20260929 18:53:05.701617 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.697721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.29.71.131:44429)
0929 18:53:05.698378 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.698383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.698392 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:05.698428 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.698430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.698445 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.698672 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:05.698725 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.699858 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.700783 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.701256 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.701363 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.701436 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.701503 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.701542 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":63,"threads_started":1}
I20260929 18:53:05.701865 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.697348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:34961 (local address 127.0.0.1:41458)
0929 18:53:05.697592 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.697604 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.697650 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.698238 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.698239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.698241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:05.698273 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.698274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.699114 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.699115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.700080 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.700082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.701631 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.701633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.701702 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.701708 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.701748 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.701799 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260929 18:53:05.702387 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.697421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.29.71.129:34961)
0929 18:53:05.698043 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.698047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.698055 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:05.698086 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.698088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.698101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.698240 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:05.698335 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.699017 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.700170 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.701521 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.702070 (+ 549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.702206 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.702237 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.702287 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":100,"threads_started":1}
I20260929 18:53:05.702440 4794 tablet_service.cc:1467] Processing CreateTablet for tablet e9f1a7c07e564177aa347b456394607d (DEFAULT_TABLE table=test-table [id=6bb5cd89652745078264e59ad4420426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:05.702696 4881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.697466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:33565 (local address 127.0.0.1:42328)
0929 18:53:05.697872 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.697882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.697926 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.698113 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.698114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.698117 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:05.698225 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.698228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.699657 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.699658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.700855 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.700859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.702359 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.702360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.702422 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.702427 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.702566 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.702607 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":67,"threads_started":1}
I20260929 18:53:05.702869 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f1a7c07e564177aa347b456394607d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:05.703033 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.697552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.29.71.130:33565)
0929 18:53:05.697702 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.697706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.697924 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:53:05.698001 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.698003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.698018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.698118 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:05.698318 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.699555 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.700980 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.702271 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.702345 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.702417 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.702600 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.702645 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260929 18:53:05.703346 4521 tablet_service.cc:1467] Processing CreateTablet for tablet e9f1a7c07e564177aa347b456394607d (DEFAULT_TABLE table=test-table [id=6bb5cd89652745078264e59ad4420426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:05.703693 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f1a7c07e564177aa347b456394607d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:05.703619 4658 tablet_service.cc:1467] Processing CreateTablet for tablet e9f1a7c07e564177aa347b456394607d (DEFAULT_TABLE table=test-table [id=6bb5cd89652745078264e59ad4420426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:05.703928 4658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f1a7c07e564177aa347b456394607d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:05.705809 4886 tablet_bootstrap.cc:492] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Bootstrap starting.
I20260929 18:53:05.706902 4886 tablet_bootstrap.cc:654] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:05.707228 4886 log.cc:824] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:05.707700 4887 tablet_bootstrap.cc:492] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Bootstrap starting.
I20260929 18:53:05.708058 4886 tablet_bootstrap.cc:492] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: No bootstrap required, opened a new log
I20260929 18:53:05.708141 4886 ts_tablet_manager.cc:1397] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:53:05.708341 4887 tablet_bootstrap.cc:654] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:05.708665 4887 log.cc:824] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:05.709024 4888 tablet_bootstrap.cc:492] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Bootstrap starting.
I20260929 18:53:05.709384 4887 tablet_bootstrap.cc:492] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: No bootstrap required, opened a new log
I20260929 18:53:05.709439 4887 ts_tablet_manager.cc:1397] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:05.710027 4886 raft_consensus.cc:348] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.710212 4886 raft_consensus.cc:374] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:05.710256 4886 raft_consensus.cc:729] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52270bceeff84e628caf71897b7e242c, State: Initialized, Role: FOLLOWER
I20260929 18:53:05.710596 4886 consensus_queue.cc:260] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.710857 4886 ts_tablet_manager.cc:1428] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:53:05.711004 4861 heartbeater.cc:503] Master 127.29.71.190:33865 was elected leader, sending a full tablet report...
I20260929 18:53:05.711748 4887 raft_consensus.cc:348] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.711902 4887 raft_consensus.cc:374] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:05.711926 4887 raft_consensus.cc:729] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a0da3e66f34b39a0a7a0fbbf0bd8c5, State: Initialized, Role: FOLLOWER
I20260929 18:53:05.712039 4887 consensus_queue.cc:260] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.712260 4887 ts_tablet_manager.cc:1428] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:53:05.712337 4725 heartbeater.cc:503] Master 127.29.71.190:33865 was elected leader, sending a full tablet report...
I20260929 18:53:05.713464 4888 tablet_bootstrap.cc:654] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:05.713702 4888 log.cc:824] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:05.714841 4888 tablet_bootstrap.cc:492] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: No bootstrap required, opened a new log
I20260929 18:53:05.714901 4888 ts_tablet_manager.cc:1397] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260929 18:53:05.716732 4888 raft_consensus.cc:348] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.716872 4888 raft_consensus.cc:374] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:05.716938 4888 raft_consensus.cc:729] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1f56ca3d184435b0e9a08383d25abf, State: Initialized, Role: FOLLOWER
I20260929 18:53:05.717061 4888 consensus_queue.cc:260] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.717358 4888 ts_tablet_manager.cc:1428] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:53:05.717388 4588 heartbeater.cc:503] Master 127.29.71.190:33865 was elected leader, sending a full tablet report...
I20260929 18:53:05.739024 4891 raft_consensus.cc:482] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:53:05.739141 4891 raft_consensus.cc:504] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.739434 4891 leader_election.cc:288] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d1f56ca3d184435b0e9a08383d25abf (127.29.71.129:34961), 27a0da3e66f34b39a0a7a0fbbf0bd8c5 (127.29.71.130:33565)
I20260929 18:53:05.743571 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.739717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:49491 (local address 127.29.71.129:34961)
0929 18:53:05.739784 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.739786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.739803 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:53:05.739859 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.739860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.739867 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.739967 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:05.740166 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.740958 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.742575 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.743153 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.743235 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.743244 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.743461 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.743502 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:53:05.743572 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.739968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:57495 (local address 127.29.71.130:33565)
0929 18:53:05.740009 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.740011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.740038 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:53:05.740067 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.740068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.740075 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.740113 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:05.740268 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.740997 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.742575 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.743154 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.743235 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.743244 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.743461 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.743501 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:53:05.743589 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.739896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:33565 (local address 127.29.71.131:57495)
0929 18:53:05.739956 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.739966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.740003 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.740136 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.740136 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.740138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:05.740203 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.740205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.741093 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.741095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.742477 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.742481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.743266 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.743267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.743365 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.743369 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.743439 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.743489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":74}
I20260929 18:53:05.743597 4866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.739642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:34961 (local address 127.29.71.131:49491)
0929 18:53:05.739702 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:05.739717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:05.739791 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:05.739997 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:05.739999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:05.740001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:05.740080 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.740082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.741093 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.741095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.742477 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:05.742481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.743266 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.743267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:05.743364 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.743369 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:53:05.743439 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:53:05.743491 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":38}
I20260929 18:53:05.744091 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f1a7c07e564177aa347b456394607d" candidate_uuid: "52270bceeff84e628caf71897b7e242c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" is_pre_election: true
I20260929 18:53:05.744282 4678 raft_consensus.cc:2454] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52270bceeff84e628caf71897b7e242c in term 0.
I20260929 18:53:05.744443 4541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f1a7c07e564177aa347b456394607d" candidate_uuid: "52270bceeff84e628caf71897b7e242c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1f56ca3d184435b0e9a08383d25abf" is_pre_election: true
I20260929 18:53:05.744529 4748 leader_election.cc:302] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a0da3e66f34b39a0a7a0fbbf0bd8c5, 52270bceeff84e628caf71897b7e242c; no voters:
I20260929 18:53:05.744618 4541 raft_consensus.cc:2454] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52270bceeff84e628caf71897b7e242c in term 0.
I20260929 18:53:05.744657 4891 raft_consensus.cc:2781] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:53:05.744709 4891 raft_consensus.cc:482] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:53:05.744745 4891 raft_consensus.cc:3037] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:05.745460 4891 raft_consensus.cc:504] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.745610 4891 leader_election.cc:288] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [CANDIDATE]: Term 1 election: Requested vote from peers 0d1f56ca3d184435b0e9a08383d25abf (127.29.71.129:34961), 27a0da3e66f34b39a0a7a0fbbf0bd8c5 (127.29.71.130:33565)
I20260929 18:53:05.745813 4678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f1a7c07e564177aa347b456394607d" candidate_uuid: "52270bceeff84e628caf71897b7e242c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5"
I20260929 18:53:05.745822 4541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f1a7c07e564177aa347b456394607d" candidate_uuid: "52270bceeff84e628caf71897b7e242c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1f56ca3d184435b0e9a08383d25abf"
I20260929 18:53:05.745910 4678 raft_consensus.cc:3037] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:05.745924 4541 raft_consensus.cc:3037] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:05.746713 4541 raft_consensus.cc:2454] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52270bceeff84e628caf71897b7e242c in term 1.
I20260929 18:53:05.746884 4748 leader_election.cc:302] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1f56ca3d184435b0e9a08383d25abf, 52270bceeff84e628caf71897b7e242c; no voters:
I20260929 18:53:05.746991 4891 raft_consensus.cc:2781] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:05.747108 4678 raft_consensus.cc:2454] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52270bceeff84e628caf71897b7e242c in term 1.
I20260929 18:53:05.747170 4891 raft_consensus.cc:686] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 1 LEADER]: Becoming Leader. State: Replica: 52270bceeff84e628caf71897b7e242c, State: Running, Role: LEADER
I20260929 18:53:05.747269 4891 consensus_queue.cc:237] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } }
I20260929 18:53:05.748164 3609 catalog_manager.cc:5495] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c reported cstate change: term changed from 0 to 1, leader changed from <none> to 52270bceeff84e628caf71897b7e242c (127.29.71.131). New cstate: current_term: 1 leader_uuid: "52270bceeff84e628caf71897b7e242c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52270bceeff84e628caf71897b7e242c" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 44429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 } health_report { overall_health: UNKNOWN } } }
I20260929 18:53:05.765209 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.762688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.29.71.131:44429)
0929 18:53:05.762750 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.762752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.762762 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:05.762842 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.762843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.762863 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.762946 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:53:05.763096 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.763572 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.764396 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.764602 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.764689 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.764877 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:53:05.764988 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:53:05.765085 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.765104 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.765124 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:53:05.767242 4678 raft_consensus.cc:1261] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 1 FOLLOWER]: Refusing update from remote peer 52270bceeff84e628caf71897b7e242c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:05.767243 4541 raft_consensus.cc:1261] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 1 FOLLOWER]: Refusing update from remote peer 52270bceeff84e628caf71897b7e242c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:05.767597 4895 consensus_queue.cc:1035] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1f56ca3d184435b0e9a08383d25abf" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:05.767753 4891 consensus_queue.cc:1035] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a0da3e66f34b39a0a7a0fbbf0bd8c5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:05.772039 4899 mvcc.cc:205] Tried to move back new op lower bound from 7334739909699731456 to 7334739909621706752. Current Snapshot: MvccSnapshot[applied={T|T < 7334739909699731456}]
I20260929 18:53:05.773175 3609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:05.773268 3609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:05.774781 3609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4bfd04aeec54be189cc56e88fce86a2: Sending DeleteTablet for 3 replicas of tablet e9f1a7c07e564177aa347b456394607d
I20260929 18:53:05.775183 4794 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f1a7c07e564177aa347b456394607d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:05 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59284
I20260929 18:53:05.775344 4658 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f1a7c07e564177aa347b456394607d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:05 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:42328
I20260929 18:53:05.775555 4908 tablet_replica.cc:330] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: stopping tablet replica
I20260929 18:53:05.775604 4521 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f1a7c07e564177aa347b456394607d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:05 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:41458
I20260929 18:53:05.775650 4908 raft_consensus.cc:2229] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:05.775756 4908 raft_consensus.cc:2258] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:05.776429 4909 tablet_replica.cc:330] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: stopping tablet replica
I20260929 18:53:05.776564 4909 raft_consensus.cc:2229] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:05.776682 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 3577
I20260929 18:53:05.777850 4908 ts_tablet_manager.cc:1905] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:05.779661 4908 ts_tablet_manager.cc:1918] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:05.779824 4908 log.cc:1196] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/e9f1a7c07e564177aa347b456394607d
I20260929 18:53:05.780189 4908 ts_tablet_manager.cc:1939] T e9f1a7c07e564177aa347b456394607d P 27a0da3e66f34b39a0a7a0fbbf0bd8c5: Deleting consensus metadata
I20260929 18:53:05.782346 4909 raft_consensus.cc:2258] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:05.782593 4907 tablet_replica.cc:330] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: stopping tablet replica
I20260929 18:53:05.782722 4907 raft_consensus.cc:2229] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:53:05.782868 4907 raft_consensus.cc:2258] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:53:05.783241 4874 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:05.773562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.29.71.188:38117)
0929 18:53:05.773838 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.773840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.773848 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:05.773873 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.773874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.773894 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.773958 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:05.774093 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.774852 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.776574 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.777221 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.777286 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.777375 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.783163 (+ 5788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34560: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":255}
W20260929 18:53:05.783401 4612 connection.cc:535] server connection from 127.0.0.1:42328 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:53:05.783833 4909 ts_tablet_manager.cc:1905] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:05.785447 4909 ts_tablet_manager.cc:1918] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:05.785573 4909 log.cc:1196] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/e9f1a7c07e564177aa347b456394607d
I20260929 18:53:05.785887 4909 ts_tablet_manager.cc:1939] T e9f1a7c07e564177aa347b456394607d P 0d1f56ca3d184435b0e9a08383d25abf: Deleting consensus metadata
W20260929 18:53:05.786302 4475 connection.cc:441] server connection from 127.0.0.1:41458 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41458 (request call id 1) could send its response
I20260929 18:53:05.786782 4907 ts_tablet_manager.cc:1905] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:05.788380 4907 ts_tablet_manager.cc:1918] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:05.788527 4907 log.cc:1196] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/e9f1a7c07e564177aa347b456394607d
I20260929 18:53:05.788833 4907 ts_tablet_manager.cc:1939] T e9f1a7c07e564177aa347b456394607d P 52270bceeff84e628caf71897b7e242c: Deleting consensus metadata
W20260929 18:53:05.789198 4748 connection.cc:441] server connection from 127.0.0.1:59284 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59284 (request call id 1) could send its response
I20260929 18:53:05.791779 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.776266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.29.71.188:38117)
0929 18:53:05.776304 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.776306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.783113 (+ 6807us) server_negotiation.cc:407] Connection header received
0929 18:53:05.783238 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.783238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.783251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.783288 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.784247 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.785015 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.786127 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.786447 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.786499 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.788284 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.788286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.788286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.788295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.790524 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.790942 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.790943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.790943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.791010 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.791208 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.791208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.791209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.791329 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.791405 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.791611 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.791683 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:53:05.794626 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.785398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.29.71.189:43121)
0929 18:53:05.785468 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.785470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.786018 (+ 548us) server_negotiation.cc:407] Connection header received
0929 18:53:05.786126 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.786127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.786142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.786196 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.787065 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.787852 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.789161 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.789502 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.789550 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.791153 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.791154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.791155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.791163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.793091 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.793503 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.793504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.793504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.793567 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.793813 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.793814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.793814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.793988 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.794099 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.794396 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.794500 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:53:05.797716 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:05.785762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.29.71.188:38117)
0929 18:53:05.785802 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:05.785804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:05.789518 (+ 3714us) server_negotiation.cc:407] Connection header received
0929 18:53:05.789628 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:05.789629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:05.789642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:05.789675 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:05.790611 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.791355 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.793037 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:05.793345 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:05.793423 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:05.795012 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:05.795014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:05.795014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:05.795021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:05.796617 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.797026 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.797027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.797027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.797081 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:05.797294 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:05.797294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:05.797295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:05.797398 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:05.797472 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:05.797595 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 18:53:05.797654 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:53:06.772279 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:06.771910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:41719)
0929 18:53:06.772092 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:06.772182 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
W20260929 18:53:06.772637 4725 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:33865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:06.772884 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:06.772634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:35923)
0929 18:53:06.772758 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:06.772831 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
W20260929 18:53:06.773214 4861 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:33865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:06.773517 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:06.773122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:52377)
0929 18:53:06.773398 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:06.773474 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":51,"threads_started":1}
W20260929 18:53:06.773753 4588 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:33865 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:07.275250 4921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bfd04aeec54be189cc56e88fce86a2)
I20260929 18:53:07.275461 4921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:07.275738 4921 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865)
I20260929 18:53:07.276158 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:07.275964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:42276)
0929 18:53:07.276070 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:07.276123 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
W20260929 18:53:07.276289 4197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:07.276726 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:07.276608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:42290)
0929 18:53:07.276670 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:07.276704 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:53:07.276853 4197 leader_election.cc:334] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865): Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:07.276924 4197 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1a1135f0b36f4ef2a7b100e4de7a9b8f; no voters: b4bfd04aeec54be189cc56e88fce86a2
I20260929 18:53:07.277053 4921 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:53:07.773249 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:07.772977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:50539)
0929 18:53:07.773100 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:07.773191 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260929 18:53:07.773726 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:07.773531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:47369)
0929 18:53:07.773650 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:07.773685 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260929 18:53:07.774261 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:07.774044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:41887)
0929 18:53:07.774182 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:07.774221 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
I20260929 18:53:08.774149 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:08.773900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:42419)
0929 18:53:08.774033 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:08.774085 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260929 18:53:08.774418 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:08.774148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:36273)
0929 18:53:08.774254 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:08.774367 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260929 18:53:08.774853 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:08.774673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:40061)
0929 18:53:08.774768 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:08.774818 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":56,"threads_started":1}
I20260929 18:53:09.333127 4930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bfd04aeec54be189cc56e88fce86a2)
I20260929 18:53:09.333240 4930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:09.333436 4930 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865)
I20260929 18:53:09.333855 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:09.333624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:57896)
0929 18:53:09.333735 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:09.333780 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260929 18:53:09.334375 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:09.334274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:57898)
0929 18:53:09.334319 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:09.334355 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:53:09.334514 4197 leader_election.cc:334] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865): Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:09.334590 4197 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1a1135f0b36f4ef2a7b100e4de7a9b8f; no voters: b4bfd04aeec54be189cc56e88fce86a2
I20260929 18:53:09.334705 4930 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:53:09.774940 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:09.774693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:44511)
0929 18:53:09.774833 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:09.774878 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260929 18:53:09.775027 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:09.774855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:50687)
0929 18:53:09.774947 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:09.774981 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":53,"threads_started":1}
I20260929 18:53:09.775381 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:09.775194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:47529)
0929 18:53:09.775304 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:09.775350 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260929 18:53:10.775622 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:10.775395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:41813)
0929 18:53:10.775517 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:10.775561 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260929 18:53:10.775806 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:10.775450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:34905)
0929 18:53:10.775652 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:10.775692 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":58,"threads_started":1}
I20260929 18:53:10.775933 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:10.775765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:36103)
0929 18:53:10.775843 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:10.775903 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":48,"threads_started":1}
I20260929 18:53:11.776304 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:11.776015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:55505)
0929 18:53:11.776210 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:11.776259 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260929 18:53:11.776456 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:11.776253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:41519)
0929 18:53:11.776380 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:11.776409 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":45,"threads_started":1}
I20260929 18:53:11.776500 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:11.776253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:58701)
0929 18:53:11.776357 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:11.776454 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260929 18:53:12.445986 4942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bfd04aeec54be189cc56e88fce86a2)
I20260929 18:53:12.446084 4942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:12.446283 4942 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865)
I20260929 18:53:12.446636 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:12.446459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:58188)
0929 18:53:12.446559 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:12.446600 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
W20260929 18:53:12.446723 4197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260929 18:53:12.447091 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:12.447000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:58196)
0929 18:53:12.447046 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:12.447072 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:53:12.447181 4197 leader_election.cc:334] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865): Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:12.447259 4197 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1a1135f0b36f4ef2a7b100e4de7a9b8f; no voters: b4bfd04aeec54be189cc56e88fce86a2
I20260929 18:53:12.447348 4942 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:53:12.777014 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:12.776804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:54207)
0929 18:53:12.776932 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:12.776969 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260929 18:53:12.777062 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:12.776754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:36097)
0929 18:53:12.776917 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:12.777000 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":61,"threads_started":1}
I20260929 18:53:12.777225 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:12.776942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:39305)
0929 18:53:12.777090 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:12.777184 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260929 18:53:13.777619 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:13.777431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:41457)
0929 18:53:13.777552 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:13.777581 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":52,"threads_started":1}
I20260929 18:53:13.777619 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:13.777413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:49829)
0929 18:53:13.777527 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:13.777569 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260929 18:53:13.777808 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:13.777539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:52535)
0929 18:53:13.777657 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:13.777763 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260929 18:53:14.778326 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:14.778075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:51355)
0929 18:53:14.778239 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:14.778275 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
I20260929 18:53:14.778326 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:14.778104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:36753)
0929 18:53:14.778239 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:14.778277 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260929 18:53:14.778450 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:14.778226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:34015)
0929 18:53:14.778347 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:14.778379 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":47,"threads_started":1}
I20260929 18:53:15.779047 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:15.778831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:52657)
0929 18:53:15.778946 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:15.778986 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":51,"threads_started":1}
I20260929 18:53:15.779048 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:15.778747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:38421)
0929 18:53:15.778917 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:15.778996 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260929 18:53:15.779083 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:15.778927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:50757)
0929 18:53:15.779012 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:15.779041 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":44,"threads_started":1}
I20260929 18:53:16.779852 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:16.779416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:45957)
0929 18:53:16.779757 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:16.779805 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":77,"threads_started":1}
I20260929 18:53:16.779855 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:16.779616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:60531)
0929 18:53:16.779764 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:16.779811 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260929 18:53:16.779855 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:16.779480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:55899)
0929 18:53:16.779757 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:16.779803 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":56,"threads_started":1}
I20260929 18:53:17.042222 4960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bfd04aeec54be189cc56e88fce86a2)
I20260929 18:53:17.042317 4960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:17.042495 4960 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865)
I20260929 18:53:17.042850 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:17.042662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:58482)
0929 18:53:17.042778 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:17.042813 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260929 18:53:17.043247 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:17.043160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:58492)
0929 18:53:17.043205 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:17.043230 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:53:17.043320 4197 leader_election.cc:334] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865): Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:17.043391 4197 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1a1135f0b36f4ef2a7b100e4de7a9b8f; no voters: b4bfd04aeec54be189cc56e88fce86a2
I20260929 18:53:17.043488 4960 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:53:17.780561 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:17.780341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:59797)
0929 18:53:17.780483 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:17.780518 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260929 18:53:17.780572 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:17.780371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:36963)
0929 18:53:17.780480 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:17.780524 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":43,"threads_started":1}
I20260929 18:53:17.780637 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:17.780469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:55939)
0929 18:53:17.780566 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:17.780596 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260929 18:53:18.781260 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:18.781010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:34163)
0929 18:53:18.781147 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:18.781197 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260929 18:53:18.781426 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:18.780957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:52291)
0929 18:53:18.781320 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:18.781370 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":71,"threads_started":1}
I20260929 18:53:18.781487 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:18.781304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:36647)
0929 18:53:18.781410 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:18.781445 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":49,"threads_started":1}
I20260929 18:53:19.782141 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:19.781866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:39093)
0929 18:53:19.782040 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:19.782083 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260929 18:53:19.782223 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:19.781918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:33169)
0929 18:53:19.782107 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:19.782142 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":56,"threads_started":1}
I20260929 18:53:19.782349 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:19.781918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:57717)
0929 18:53:19.782274 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:19.782309 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":63,"threads_started":1}
I20260929 18:53:20.782990 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:20.782766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:33537)
0929 18:53:20.782891 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:20.782930 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260929 18:53:20.783211 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:20.783037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:51645)
0929 18:53:20.783129 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:20.783162 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
I20260929 18:53:20.783267 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:20.782803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:53827)
0929 18:53:20.783142 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:20.783225 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":53,"threads_started":1}
I20260929 18:53:21.783918 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:21.783695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:59723)
0929 18:53:21.783841 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:21.783875 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260929 18:53:21.784056 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:21.783753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:55991)
0929 18:53:21.783993 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:21.784022 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
I20260929 18:53:21.784054 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:21.783753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:55343)
0929 18:53:21.783961 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:21.784006 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":54,"threads_started":1}
I20260929 18:53:22.784685 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:22.784455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:37341)
0929 18:53:22.784593 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:22.784633 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260929 18:53:22.784687 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:22.784477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:49087)
0929 18:53:22.784622 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:22.784652 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":53,"threads_started":1}
I20260929 18:53:22.784688 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:22.784477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:51249)
0929 18:53:22.784590 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:22.784630 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260929 18:53:23.785313 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:23.785096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:56103)
0929 18:53:23.785209 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:23.785254 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260929 18:53:23.785329 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:23.785051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:46893)
0929 18:53:23.785218 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:23.785258 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260929 18:53:23.785426 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:23.785149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:55167)
0929 18:53:23.785349 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:23.785381 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":65,"threads_started":1}
I20260929 18:53:23.825330 4984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bfd04aeec54be189cc56e88fce86a2)
I20260929 18:53:23.825434 4984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:23.825651 4984 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865)
I20260929 18:53:23.826054 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:23.825868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:35320)
0929 18:53:23.825981 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:23.826022 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
W20260929 18:53:23.826143 4197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260929 18:53:23.826623 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:23.826551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:35326)
0929 18:53:23.826582 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:23.826607 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260929 18:53:23.826695 4197 leader_election.cc:334] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865): Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:23.826746 4197 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1a1135f0b36f4ef2a7b100e4de7a9b8f; no voters: b4bfd04aeec54be189cc56e88fce86a2
I20260929 18:53:23.826835 4984 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:53:24.785971 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:24.785776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:39059)
0929 18:53:24.785877 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:24.785915 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":44,"threads_started":1}
I20260929 18:53:24.785975 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:24.785750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:40667)
0929 18:53:24.785884 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:24.785922 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260929 18:53:24.786051 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:24.785758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:39479)
0929 18:53:24.785880 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:24.786010 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260929 18:53:25.786633 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:25.786473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:47495)
0929 18:53:25.786572 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:25.786595 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":43,"threads_started":1}
I20260929 18:53:25.786613 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:25.786363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:57537)
0929 18:53:25.786511 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:25.786553 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260929 18:53:25.786748 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:25.786502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:55567)
0929 18:53:25.786645 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:25.786696 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
I20260929 18:53:26.787235 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:26.786992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:36269)
0929 18:53:26.787127 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:26.787170 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260929 18:53:26.787303 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:26.787090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:56567)
0929 18:53:26.787214 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:26.787250 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260929 18:53:26.787390 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:26.787231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:54795)
0929 18:53:26.787321 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:26.787349 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":40,"threads_started":1}
I20260929 18:53:27.787858 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:27.787606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:55059)
0929 18:53:27.787748 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:27.787792 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260929 18:53:27.787935 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:27.787679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:56809)
0929 18:53:27.787852 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:27.787887 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":51,"threads_started":1}
I20260929 18:53:27.787966 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:27.787806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:60389)
0929 18:53:27.787898 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:27.787924 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
I20260929 18:53:28.788678 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:28.788424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:38629)
0929 18:53:28.788573 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:28.788617 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260929 18:53:28.788697 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:28.788512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:35787)
0929 18:53:28.788610 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:28.788649 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
I20260929 18:53:28.788744 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:28.788573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:32959)
0929 18:53:28.788678 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:28.788704 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260929 18:53:28.805145 5002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bfd04aeec54be189cc56e88fce86a2)
I20260929 18:53:28.805249 5002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:28.805467 5002 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865)
I20260929 18:53:28.805817 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:28.805606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:48650)
0929 18:53:28.805731 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:28.805779 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260929 18:53:28.806336 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:28.806235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:48666)
0929 18:53:28.806281 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:28.806314 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 18:53:28.806413 4197 leader_election.cc:334] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865): Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:28.806458 4197 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1a1135f0b36f4ef2a7b100e4de7a9b8f; no voters: b4bfd04aeec54be189cc56e88fce86a2
I20260929 18:53:28.806527 5002 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:53:29.789321 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:29.789091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:59591)
0929 18:53:29.789228 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:29.789267 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260929 18:53:29.789376 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:29.789183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:44589)
0929 18:53:29.789282 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:29.789322 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
I20260929 18:53:29.789537 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:29.789304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:60399)
0929 18:53:29.789463 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:29.789507 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":51,"threads_started":1}
I20260929 18:53:30.789990 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:30.789731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:40283)
0929 18:53:30.789891 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:30.789932 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260929 18:53:30.790119 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:30.789916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:40741)
0929 18:53:30.790033 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:30.790066 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
I20260929 18:53:30.790191 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:30.789869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:53359)
0929 18:53:30.790027 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:30.790056 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":43,"threads_started":1}
I20260929 18:53:31.790709 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:31.790493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:57809)
0929 18:53:31.790613 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:31.790652 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260929 18:53:31.790802 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:31.790612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:43017)
0929 18:53:31.790715 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:31.790753 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
I20260929 18:53:31.790864 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:31.790581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:34973)
0929 18:53:31.790790 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:31.790825 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":54,"threads_started":1}
I20260929 18:53:32.791442 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:32.791205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:57849)
0929 18:53:32.791333 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:32.791378 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260929 18:53:32.791515 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:32.791332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:33087)
0929 18:53:32.791447 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:32.791475 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260929 18:53:32.791756 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:32.791553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:50091)
0929 18:53:32.791677 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:32.791720 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":47,"threads_started":1}
I20260929 18:53:33.166460 5017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4bfd04aeec54be189cc56e88fce86a2)
I20260929 18:53:33.166586 5017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4bfd04aeec54be189cc56e88fce86a2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33865 } } peers { permanent_uuid: "1a1135f0b36f4ef2a7b100e4de7a9b8f" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 43121 } attrs { promote: false } } peers { permanent_uuid: "b3ed090adf8c423bab5446613b820926" member_type: NON_VOTER last_known_addr { host: "127.29.71.188" port: 38117 } attrs { promote: true } }
I20260929 18:53:33.166769 5017 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865)
I20260929 18:53:33.167092 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:33.166937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:48852)
0929 18:53:33.167023 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:33.167060 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":53,"threads_started":1}
W20260929 18:53:33.167173 4197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260929 18:53:33.167644 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:33.167463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.0.0.1:48856)
0929 18:53:33.167511 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:33.167589 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:53:33.167728 4197 leader_election.cc:334] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b4bfd04aeec54be189cc56e88fce86a2 (127.29.71.190:33865): Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
I20260929 18:53:33.167771 4197 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1a1135f0b36f4ef2a7b100e4de7a9b8f; no voters: b4bfd04aeec54be189cc56e88fce86a2
I20260929 18:53:33.167831 5017 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a1135f0b36f4ef2a7b100e4de7a9b8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:53:33.792202 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:33.791950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:41695)
0929 18:53:33.792096 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:33.792136 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260929 18:53:33.792408 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:33.792229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:43829)
0929 18:53:33.792332 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:33.792368 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260929 18:53:33.792393 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:33.792036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:36063)
0929 18:53:33.792302 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:33.792341 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":54,"threads_started":1}
I20260929 18:53:34.793136 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:34.792802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:50895)
0929 18:53:34.793035 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:34.793079 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":60,"threads_started":1}
I20260929 18:53:34.793174 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:34.792802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:50171)
0929 18:53:34.793099 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:34.793129 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":67,"threads_started":1}
I20260929 18:53:34.793174 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:34.792867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:56259)
0929 18:53:34.793075 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:34.793118 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260929 18:53:35.793993 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:35.793588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.130:43769)
0929 18:53:35.793839 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:35.793889 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":61,"threads_started":1}
I20260929 18:53:35.794142 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:35.793658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.129:51039)
0929 18:53:35.793998 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:35.794039 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":59,"threads_started":1}
I20260929 18:53:35.794464 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:35.794187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33865 (local address 127.29.71.131:36383)
0929 18:53:35.794299 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:35.794330 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":44,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.29.71.190:33865,127.29.71.189:43121,127.29.71.188:38117, num_attempts: 1)
I20260929 18:53:35.978009 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 4370
I20260929 18:53:35.982862 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 4595
I20260929 18:53:35.987406 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 4731
I20260929 18:53:35.992385 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 4073
I20260929 18:53:35.996860 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 4267
2026-09-29T18:53:35Z chronyd exiting
I20260929 18:53:36.012916 29982 test_util.cc:175] -----------------------------------------------
I20260929 18:53:36.013003 29982 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790707954762284-29982-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33180 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:53:36 dist-test-slave-q2kh krb5kdc[5031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:53:36 dist-test-slave-q2kh krb5kdc[5031](info): set up 2 sockets
Sep 29 18:53:36 dist-test-slave-q2kh krb5kdc[5031](info): commencing operation
krb5kdc: starting...
W20260929 18:53:37.062674 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab.
Entry for principal oryx/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab.
I20260929 18:53:37.189771 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:43919
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab
--principal=oryx/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:37.271430 5044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:37.271701 5044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:37.271739 5044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:37.272935 5044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:37.273010 5044 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:53:37.273032 5044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:37.273053 5044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:37.273075 5044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:37.273102 5044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:37.274638 5044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:43919
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.190.keytab
--principal=oryx/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:37.274914 5044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:37.275151 5044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:37.280853 5050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.280860 5049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.280860 5052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:37.280923 5044 server_base.cc:1034] running on GCE node
I20260929 18:53:37.281423 5044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:37.281745 5044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:37.282891 5044 hybrid_clock.cc:648] HybridClock initialized: now 1790708017282870 us; error 40 us; skew 500 ppm
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:37.285398 5044 init.cc:377] Logged in from keytab as oryx/127.29.71.190@KRBTEST.COM (short username oryx)
I20260929 18:53:37.285868 5044 webserver.cc:460] Webserver started at http://127.29.71.190:39507/ using document root <none> and password file <none>
I20260929 18:53:37.286060 5044 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:37.286101 5044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:37.286254 5044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:37.287304 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "aeb2995c43bb43c0af5549e6317b368c"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "b8b1352abd1d9874817012685c274213"
server_key_iv: "69cad579b215a04419929a40606d25cf"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.287621 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "aeb2995c43bb43c0af5549e6317b368c"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "b8b1352abd1d9874817012685c274213"
server_key_iv: "69cad579b215a04419929a40606d25cf"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.289031 5044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:37.289831 5059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.290076 5044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:37.290233 5044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "aeb2995c43bb43c0af5549e6317b368c"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "b8b1352abd1d9874817012685c274213"
server_key_iv: "69cad579b215a04419929a40606d25cf"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.290328 5044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:37.303268 5044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:37.305547 5044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:37.305732 5044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:37.310751 5044 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:43919
I20260929 18:53:37.310793 5121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:43919 every 8 connection(s)
I20260929 18:53:37.311146 5044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:37.311795 5122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:37.314086 5122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Bootstrap starting.
I20260929 18:53:37.314654 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 5044
I20260929 18:53:37.314759 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:37.314824 5122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:37.314985 29982 external_mini_cluster.cc:1428] Setting key 929b1f009737b25eab5a3842760d6839
I20260929 18:53:37.315169 5122 log.cc:824] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:37.315886 5122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: No bootstrap required, opened a new log
I20260929 18:53:37.317806 5122 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb2995c43bb43c0af5549e6317b368c" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43919 } }
I20260929 18:53:37.317947 5122 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:37.317970 5122 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb2995c43bb43c0af5549e6317b368c, State: Initialized, Role: FOLLOWER
I20260929 18:53:37.318123 5122 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb2995c43bb43c0af5549e6317b368c" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43919 } }
I20260929 18:53:37.318269 5122 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:37.318316 5122 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:37.318373 5122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:37.318872 5122 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb2995c43bb43c0af5549e6317b368c" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43919 } }
I20260929 18:53:37.318985 5122 leader_election.cc:302] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeb2995c43bb43c0af5549e6317b368c; no voters:
I20260929 18:53:37.319180 5122 leader_election.cc:288] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:37.319201 5127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:37.319305 5127 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [term 1 LEADER]: Becoming Leader. State: Replica: aeb2995c43bb43c0af5549e6317b368c, State: Running, Role: LEADER
I20260929 18:53:37.319463 5122 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:37.319440 5127 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb2995c43bb43c0af5549e6317b368c" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43919 } }
I20260929 18:53:37.319839 5128 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeb2995c43bb43c0af5549e6317b368c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb2995c43bb43c0af5549e6317b368c" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43919 } } }
I20260929 18:53:37.319941 5128 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:37.319839 5127 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeb2995c43bb43c0af5549e6317b368c. Latest consensus state: current_term: 1 leader_uuid: "aeb2995c43bb43c0af5549e6317b368c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeb2995c43bb43c0af5549e6317b368c" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43919 } } }
I20260929 18:53:37.320348 5127 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:37.320866 5134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:37.321502 5134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:37.323073 5134 catalog_manager.cc:1347] Generated new cluster ID: bdb17e4cdb1241f88413dfe198926311
I20260929 18:53:37.323124 5134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:53:37.328586 5134 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:37.329258 5134 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:37.329999 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.315506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.29.71.190:43919)
0929 18:53:37.315654 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.315664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.315699 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:53:37.316183 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.316190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.316238 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.316383 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.317077 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.317994 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.318887 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.319529 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.319930 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.322529 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.322534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.322534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.322546 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.328317 (+ 5771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.328752 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.328755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.328755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.328858 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.329085 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.329086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.329086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.329341 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.329501 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.329635 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.329769 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260929 18:53:37.333810 5134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Generated new TSK 0
I20260929 18:53:37.333972 5134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab.
Entry for principal oryx/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab.
I20260929 18:53:37.361447 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43919
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab
--principal=oryx/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:37.445556 5148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:37.445793 5148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:37.445835 5148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:37.447091 5148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:37.447160 5148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:37.447197 5148 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:53:37.447256 5148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:37.448799 5148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab
--principal=oryx/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:37.449077 5148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:37.449313 5148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:37.454571 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.454586 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.454569 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:37.454617 5148 server_base.cc:1034] running on GCE node
I20260929 18:53:37.455055 5148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:37.455330 5148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:37.456499 5148 hybrid_clock.cc:648] HybridClock initialized: now 1790708017456488 us; error 41 us; skew 500 ppm
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:37.459137 5148 init.cc:377] Logged in from keytab as oryx/127.29.71.129@KRBTEST.COM (short username oryx)
I20260929 18:53:37.459587 5148 webserver.cc:460] Webserver started at http://127.29.71.129:34373/ using document root <none> and password file <none>
I20260929 18:53:37.459821 5148 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:37.459899 5148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:37.460053 5148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:37.461124 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "a2f5bd0bcc694c388151d8c591f9a42a"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "394aac24b7c91c39f5efa24b0474de35"
server_key_iv: "f77c2f60cd4601524f0a0c7365d8d10c"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.461493 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "a2f5bd0bcc694c388151d8c591f9a42a"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "394aac24b7c91c39f5efa24b0474de35"
server_key_iv: "f77c2f60cd4601524f0a0c7365d8d10c"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.462955 5148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:37.463724 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.463930 5148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:37.464053 5148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "a2f5bd0bcc694c388151d8c591f9a42a"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "394aac24b7c91c39f5efa24b0474de35"
server_key_iv: "f77c2f60cd4601524f0a0c7365d8d10c"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.464139 5148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:37.474992 5148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:37.477010 5148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:37.477174 5148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:37.477694 5148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:37.477758 5148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.477792 5148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:37.477833 5148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:37.485008 5148 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:36297
I20260929 18:53:37.485055 5278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:36297 every 8 connection(s)
I20260929 18:53:37.485399 5148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:53:37.486845 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 5148
I20260929 18:53:37.486981 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:53:37.487186 29982 external_mini_cluster.cc:1428] Setting key 1360860e9de33613dfc588612e5ef41f
I20260929 18:53:37.487784 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.478144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:35041 (local address 127.29.71.190:43919)
0929 18:53:37.478223 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.478225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.478435 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:53:37.478945 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.478946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.478959 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.479043 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.479731 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.480250 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.481244 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.481479 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.481565 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.484642 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.484643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.484644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.484652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.486118 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.486579 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.486581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.486581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.486639 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.486952 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.486953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.486953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.487090 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.487182 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.487605 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.487685 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:53:37.487919 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.478129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43919 (local address 127.29.71.129:35041)
0929 18:53:37.478290 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.478330 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.478806 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.479144 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.479148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.479448 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:37.479634 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.479639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.480347 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.480349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.481153 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.481155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.481575 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.481576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.481701 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.482220 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:37.482229 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:37.484544 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:37.486243 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.486250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.486252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.486492 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.486761 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.486762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.486762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.486872 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.487172 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:37.487178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:37.487357 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.487476 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.487679 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260929 18:53:37.488401 5279 heartbeater.cc:347] Connected to a master server at 127.29.71.190:43919
I20260929 18:53:37.488511 5279 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:37.488722 5279 heartbeater.cc:511] Master 127.29.71.190:43919 requested a full tablet report, sending...
I20260929 18:53:37.489292 5076 ts_manager.cc:194] Registered new tserver with Master: a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297)
I20260929 18:53:37.490264 5076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:35041
WARNING: no policy specified for oryx/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.130@KRBTEST.COM" created.
I20260929 18:53:37.499838 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.489457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:55211 (local address 127.29.71.190:43919)
0929 18:53:37.489493 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.489495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.494382 (+ 4887us) server_negotiation.cc:407] Connection header received
0929 18:53:37.494457 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.494458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.494470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.494538 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.494839 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.495360 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.496110 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.496287 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.496346 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.497273 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.497274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.497274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.497280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.498715 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.499072 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.499073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.499074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.499129 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.499321 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.499322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.499322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.499432 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.499507 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.499675 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.499742 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:53:37.499836 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.489408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43919 (local address 127.29.71.129:55211)
0929 18:53:37.494310 (+ 4902us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.494321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.494397 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.494543 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.494544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.494709 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:37.494760 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.494762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.495459 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.495460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.496015 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.496017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.496357 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.496358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.496409 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.496796 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:37.496800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:37.497195 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:37.498815 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.498816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.498816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.498987 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.499196 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.499196 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.499197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.499257 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.499500 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:37.499501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:37.499545 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.499626 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.499734 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4878,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab.
Entry for principal oryx/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab.
I20260929 18:53:37.513006 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43919
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab
--principal=oryx/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:37.593990 5288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:37.594201 5288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:37.594226 5288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:37.595293 5288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:37.595326 5288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:37.595346 5288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:53:37.595363 5288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:37.596784 5288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab
--principal=oryx/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:37.596964 5288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:37.597138 5288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:37.601984 5294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.601992 5293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.601997 5296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:37.602392 5288 server_base.cc:1034] running on GCE node
I20260929 18:53:37.602654 5288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:37.602897 5288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:37.604038 5288 hybrid_clock.cc:648] HybridClock initialized: now 1790708017604018 us; error 37 us; skew 500 ppm
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:37.606652 5288 init.cc:377] Logged in from keytab as oryx/127.29.71.130@KRBTEST.COM (short username oryx)
I20260929 18:53:37.607088 5288 webserver.cc:460] Webserver started at http://127.29.71.130:33653/ using document root <none> and password file <none>
I20260929 18:53:37.607337 5288 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:37.607384 5288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:37.607547 5288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:37.608641 5288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "e5965c72f56c4b848f748e9dcae760c7"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "40aa8f30f822451b36723e486c145148"
server_key_iv: "c652123e6477c4b3f1d5bb665b91aee1"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.608996 5288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "e5965c72f56c4b848f748e9dcae760c7"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "40aa8f30f822451b36723e486c145148"
server_key_iv: "c652123e6477c4b3f1d5bb665b91aee1"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.610424 5288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:53:37.611245 5303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.611428 5288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:37.611526 5288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "e5965c72f56c4b848f748e9dcae760c7"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "40aa8f30f822451b36723e486c145148"
server_key_iv: "c652123e6477c4b3f1d5bb665b91aee1"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.611614 5288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:37.640425 5288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:37.642640 5288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:37.642827 5288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:37.643365 5288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:37.643437 5288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.643474 5288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:37.643503 5288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.650429 5288 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:38645
I20260929 18:53:37.650504 5418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:38645 every 8 connection(s)
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:37.650885 5288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:37.653863 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.643898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:55947 (local address 127.29.71.190:43919)
0929 18:53:37.643998 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.644000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.644248 (+ 248us) server_negotiation.cc:407] Connection header received
0929 18:53:37.644799 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.644801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.644813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.644869 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.645753 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.646615 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.648326 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.648589 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.648648 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.651402 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.651403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.651404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.651410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.652815 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.653157 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.653158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.653158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.653206 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.653403 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.653404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.653404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.653498 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.653548 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.653746 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.653801 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:53:37.654089 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.643938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43919 (local address 127.29.71.130:55947)
0929 18:53:37.644124 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.644161 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.644700 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.644974 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.644979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.645364 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:37.645653 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.645660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.646731 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.646732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.648233 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.648237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.648689 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.648690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.648868 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.649462 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:37.649473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:37.651302 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:37.652886 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.652891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.652897 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.653094 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.653264 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.653264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.653264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.653352 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.653552 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:37.653556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:37.653659 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.653737 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.653845 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260929 18:53:37.654587 5419 heartbeater.cc:347] Connected to a master server at 127.29.71.190:43919
I20260929 18:53:37.654680 5419 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:37.654853 5419 heartbeater.cc:511] Master 127.29.71.190:43919 requested a full tablet report, sending...
I20260929 18:53:37.655259 5075 ts_manager.cc:194] Registered new tserver with Master: e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645)
I20260929 18:53:37.655699 5075 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:55947
I20260929 18:53:37.658353 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 5288
I20260929 18:53:37.658470 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:53:37.658671 29982 external_mini_cluster.cc:1428] Setting key 6a80a51ad2086f311c581462463e7b62
I20260929 18:53:37.661077 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.655646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:54837 (local address 127.29.71.190:43919)
0929 18:53:37.655684 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.655686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.655762 (+ 76us) server_negotiation.cc:407] Connection header received
0929 18:53:37.655881 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.655881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.655891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.655941 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.656282 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.656737 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.657375 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.657620 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.657672 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.658520 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.658522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.658522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.658528 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.659961 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.660292 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.660293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.660294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.660345 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.660558 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.660558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.660559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.660699 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.660776 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.660921 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.660991 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:53:37.661098 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.655572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43919 (local address 127.29.71.130:54837)
0929 18:53:37.655701 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.655712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.655776 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.655944 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.655945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.656147 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:37.656208 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.656210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.656819 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.656820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.657298 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.657299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.657683 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.657684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.657728 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.658078 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:37.658081 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:37.658417 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:37.660048 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.660049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.660050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.660211 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.660434 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.660435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.660435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.660489 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.660768 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:37.660769 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:37.660813 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.660903 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.660991 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab.
Entry for principal oryx/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab.
I20260929 18:53:37.683336 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43919
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab
--principal=oryx/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:37.764998 5428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:37.765203 5428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:37.765228 5428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:37.766469 5428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:37.766523 5428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:37.766546 5428 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:53:37.766587 5428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:37.768226 5428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab
--principal=oryx/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.71.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:37.768483 5428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:37.768736 5428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:37.773729 5433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.773729 5434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:37.773779 5436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:37.773964 5428 server_base.cc:1034] running on GCE node
I20260929 18:53:37.774216 5428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:37.774471 5428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:37.775648 5428 hybrid_clock.cc:648] HybridClock initialized: now 1790708017775631 us; error 46 us; skew 500 ppm
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:37.778066 5428 init.cc:377] Logged in from keytab as oryx/127.29.71.131@KRBTEST.COM (short username oryx)
I20260929 18:53:37.778577 5428 webserver.cc:460] Webserver started at http://127.29.71.131:38071/ using document root <none> and password file <none>
I20260929 18:53:37.778829 5428 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:37.778898 5428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:37.779050 5428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:37.780108 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "e29d8e10980387230315edd71a3d48cc"
server_key_iv: "c1e9d64350a0cf1bafeef90530900b8a"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.780478 5428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "e29d8e10980387230315edd71a3d48cc"
server_key_iv: "c1e9d64350a0cf1bafeef90530900b8a"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.781944 5428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 18:53:37.782871 5443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.783094 5428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:53:37.783200 5428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
format_stamp: "Formatted at 2026-09-29 18:53:37 on dist-test-slave-q2kh"
server_key: "e29d8e10980387230315edd71a3d48cc"
server_key_iv: "c1e9d64350a0cf1bafeef90530900b8a"
server_key_version: "encryptionkey@0"
I20260929 18:53:37.783284 5428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:37.804152 5428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:37.806399 5428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:37.806572 5428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:37.807133 5428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:37.807199 5428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:37.807245 5428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:37.807273 5428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:37.814555 5428 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:34629
I20260929 18:53:37.814881 5558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:34629 every 8 connection(s)
I20260929 18:53:37.814929 5428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:53:37.817005 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.807638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:43839 (local address 127.29.71.190:43919)
0929 18:53:37.807693 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.807695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.807899 (+ 204us) server_negotiation.cc:407] Connection header received
0929 18:53:37.808408 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.808409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.808421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.808469 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.809078 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.809573 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.810685 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.810893 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.810986 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.814258 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.814259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.814260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.814267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.815892 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.816268 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.816268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.816269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.816319 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.816491 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.816491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.816491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.816603 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.816650 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.816882 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.816941 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:53:37.817194 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.807639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43919 (local address 127.29.71.131:43839)
0929 18:53:37.807786 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.807819 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.808304 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.808549 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.808552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.808783 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:37.808975 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.808980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.809665 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.809666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.810594 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.810597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.810994 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.810995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.811145 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.811691 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:37.811702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:37.814141 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:37.815966 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.815970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.815972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.816198 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.816361 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.816361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.816361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.816442 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.816656 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:37.816660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:37.816769 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.816853 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.816975 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260929 18:53:37.817662 5559 heartbeater.cc:347] Connected to a master server at 127.29.71.190:43919
I20260929 18:53:37.817767 5559 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:37.817896 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 5428
I20260929 18:53:37.818004 5559 heartbeater.cc:511] Master 127.29.71.190:43919 requested a full tablet report, sending...
I20260929 18:53:37.818027 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:53:37.818284 29982 external_mini_cluster.cc:1428] Setting key c8b7a43ab229ad09293fc7fd301762e6
I20260929 18:53:37.818420 5075 ts_manager.cc:194] Registered new tserver with Master: e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:37.818941 5075 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:43839
I20260929 18:53:37.819131 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:53:37.825428 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.818761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:39397 (local address 127.29.71.190:43919)
0929 18:53:37.818802 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.818804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.819319 (+ 515us) server_negotiation.cc:407] Connection header received
0929 18:53:37.819379 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.819380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.819390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.819445 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.819824 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.820425 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.821553 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.821756 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.821830 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.822768 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.822770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.822770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.822777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.824335 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.824732 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.824732 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.824733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.824781 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.824982 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.824983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.824983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.825107 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.825159 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.825315 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.825367 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:53:37.825549 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.818802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43919 (local address 127.29.71.131:39397)
0929 18:53:37.819262 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.819275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.819332 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.819450 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.819451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.819685 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:37.819750 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.819752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.820533 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.820535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.821448 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.821451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.821828 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.821829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.821893 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.822390 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:37.822395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:37.822686 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:37.824425 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.824426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.824427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.824656 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.824844 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:37.824845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:37.824845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:37.824918 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:37.825170 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:37.825171 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:37.825221 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.825300 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.825417 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":64,"threads_started":1}
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:53:37.841586 5126 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:37.836325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.29.71.190:43919)
0929 18:53:37.836386 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.836387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.836406 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:53:37.836431 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.836431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.836440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.836490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.836995 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.837471 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.838209 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.838403 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.838454 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.841459 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37588: BlockingRecv error: recv got EOF from 127.0.0.1:37588 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 18:53:37 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:37.849128 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.842286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.29.71.190:43919)
0929 18:53:37.842355 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.842357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.842440 (+ 83us) server_negotiation.cc:407] Connection header received
0929 18:53:37.842530 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.842531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.842540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.842593 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:37.843069 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.843542 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.844244 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.844442 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.844524 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.846616 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:37.846617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:37.846618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:37.846625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:37.847937 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.848310 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.848312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.848312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.848371 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:37.848573 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:37.848574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:37.848574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:37.848709 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:37.848821 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.848940 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.849021 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:53:37.850737 5075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:53:37.852547 5075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:53:37.859298 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.855016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:34629 (local address 127.0.0.1:46102)
0929 18:53:37.855682 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.855692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.855764 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.855987 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.855988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.855989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:37.856025 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.856026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.856855 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.856856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.858298 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.858301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.858989 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.858990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.859069 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.859084 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.859150 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.859205 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":74,"thread_start_us":89,"threads_started":1}
I20260929 18:53:37.859407 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.855152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.29.71.130:38645)
0929 18:53:37.855325 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.855331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.855341 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:37.855382 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.855384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.855399 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.855510 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:37.855654 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.856805 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.858393 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.859075 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.859145 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.859209 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.859274 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.859308 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260929 18:53:37.859498 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.854959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.29.71.129:36297)
0929 18:53:37.855395 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.855401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.855412 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:53:37.855454 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.855456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.855472 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.855577 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:37.855955 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.856992 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.858222 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.858944 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.859028 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.859102 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.859351 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.859391 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":71,"threads_started":1}
I20260929 18:53:37.859789 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.855217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.29.71.131:34629)
0929 18:53:37.855411 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:37.855415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:37.855760 (+ 345us) server_negotiation.cc:407] Connection header received
0929 18:53:37.855833 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:37.855835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:37.855850 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:37.855923 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:37.856097 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.856768 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.858415 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.858892 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.858978 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.859057 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:37.859685 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 18:53:37.859720 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260929 18:53:37.859417 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.854889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:38645 (local address 127.0.0.1:54190)
0929 18:53:37.855116 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.855135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.855201 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.855487 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.855491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.855493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:37.855571 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.855573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.856904 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.856905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.858315 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.858317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.859144 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.859145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.859208 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.859211 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.859263 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.859295 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":103,"thread_start_us":80,"threads_started":1}
I20260929 18:53:37.859498 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:37.854811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36297 (local address 127.0.0.1:35652)
0929 18:53:37.854915 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:37.854960 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:37.855071 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:37.855830 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:37.855831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:37.855832 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:37.855883 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.855885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.857076 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.857077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.858088 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:37.858090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:37.859140 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:37.859141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:37.859214 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:37.859219 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:53:37.859268 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:53:37.859309 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260929 18:53:37.860556 5353 tablet_service.cc:1467] Processing CreateTablet for tablet dcb84878dd2c4a2f9154f280634e4f91 (DEFAULT_TABLE table=test-table [id=cb8231562354415ba7ca41e900006b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:37.860554 5493 tablet_service.cc:1467] Processing CreateTablet for tablet dcb84878dd2c4a2f9154f280634e4f91 (DEFAULT_TABLE table=test-table [id=cb8231562354415ba7ca41e900006b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:37.860929 5493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb84878dd2c4a2f9154f280634e4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:37.860929 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb84878dd2c4a2f9154f280634e4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:37.864780 5587 tablet_bootstrap.cc:492] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Bootstrap starting.
I20260929 18:53:37.864888 5588 tablet_bootstrap.cc:492] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Bootstrap starting.
I20260929 18:53:37.865579 5588 tablet_bootstrap.cc:654] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:37.865604 5587 tablet_bootstrap.cc:654] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:37.865583 5213 tablet_service.cc:1467] Processing CreateTablet for tablet dcb84878dd2c4a2f9154f280634e4f91 (DEFAULT_TABLE table=test-table [id=cb8231562354415ba7ca41e900006b9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:37.865938 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb84878dd2c4a2f9154f280634e4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:37.865933 5587 log.cc:824] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:37.865927 5588 log.cc:824] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:37.866654 5587 tablet_bootstrap.cc:492] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: No bootstrap required, opened a new log
I20260929 18:53:37.866662 5588 tablet_bootstrap.cc:492] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: No bootstrap required, opened a new log
I20260929 18:53:37.866736 5588 ts_tablet_manager.cc:1397] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:37.866737 5587 ts_tablet_manager.cc:1397] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:37.868184 5587 raft_consensus.cc:348] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:37.868300 5587 raft_consensus.cc:374] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:37.868327 5587 raft_consensus.cc:729] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30af67ffb884326b3d2f4f3cdf2e2a1, State: Initialized, Role: FOLLOWER
I20260929 18:53:37.868443 5587 consensus_queue.cc:260] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:37.868798 5587 ts_tablet_manager.cc:1428] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:53:37.868775 5588 raft_consensus.cc:348] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:37.868922 5588 raft_consensus.cc:374] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:37.868923 5559 heartbeater.cc:503] Master 127.29.71.190:43919 was elected leader, sending a full tablet report...
I20260929 18:53:37.868953 5588 raft_consensus.cc:729] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5965c72f56c4b848f748e9dcae760c7, State: Initialized, Role: FOLLOWER
I20260929 18:53:37.869056 5588 consensus_queue.cc:260] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:37.869300 5588 ts_tablet_manager.cc:1428] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:53:37.869444 5419 heartbeater.cc:503] Master 127.29.71.190:43919 was elected leader, sending a full tablet report...
I20260929 18:53:37.869854 5593 tablet_bootstrap.cc:492] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Bootstrap starting.
I20260929 18:53:37.870503 5593 tablet_bootstrap.cc:654] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:37.870801 5593 log.cc:824] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:37.871519 5593 tablet_bootstrap.cc:492] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: No bootstrap required, opened a new log
I20260929 18:53:37.871572 5593 ts_tablet_manager.cc:1397] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:37.873044 5593 raft_consensus.cc:348] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:37.873227 5593 raft_consensus.cc:374] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:37.873286 5593 raft_consensus.cc:729] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Initialized, Role: FOLLOWER
I20260929 18:53:37.873410 5593 consensus_queue.cc:260] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:37.873672 5593 ts_tablet_manager.cc:1428] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:53:37.873696 5279 heartbeater.cc:503] Master 127.29.71.190:43919 was elected leader, sending a full tablet report...
W20260929 18:53:37.901472 5420 tablet.cc:2369] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:53:37.986264 5280 tablet.cc:2369] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:53:38.022241 5592 raft_consensus.cc:482] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:53:38.022415 5592 raft_consensus.cc:504] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:38.022820 5592 leader_election.cc:288] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.025768 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.023119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:41655 (local address 127.29.71.129:36297)
0929 18:53:38.023189 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.023192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.023204 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:53:38.023258 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.023259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.023269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.023332 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:38.023518 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.023998 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.025000 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.025364 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.025460 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.025469 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.025655 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.025693 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:53:38.025785 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.023050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36297 (local address 127.29.71.130:41655)
0929 18:53:38.023111 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.023124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.023190 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.023379 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.023381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.023383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:38.023457 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.023459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.024093 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.024095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.024888 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.024890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.025460 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.025461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.025567 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.025570 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.025654 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.025697 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:53:38.025880 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.023173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:34629 (local address 127.29.71.130:43449)
0929 18:53:38.023355 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.023365 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.023410 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.023556 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.023557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.023559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:38.023632 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.023634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.024265 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.024266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.025023 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.025024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.025583 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.025584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.025655 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.025659 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.025745 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.025785 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260929 18:53:38.025857 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.023365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:43449 (local address 127.29.71.131:34629)
0929 18:53:38.023406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.023408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.023417 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:38.023473 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.023474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.023480 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.023550 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:38.023720 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.024181 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.025128 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.025502 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.025585 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.025593 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.025752 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.025787 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:53:38.026189 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb84878dd2c4a2f9154f280634e4f91" candidate_uuid: "e5965c72f56c4b848f748e9dcae760c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" is_pre_election: true
I20260929 18:53:38.026199 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb84878dd2c4a2f9154f280634e4f91" candidate_uuid: "e5965c72f56c4b848f748e9dcae760c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" is_pre_election: true
I20260929 18:53:38.026358 5233 raft_consensus.cc:2454] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5965c72f56c4b848f748e9dcae760c7 in term 0.
I20260929 18:53:38.026360 5513 raft_consensus.cc:2454] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5965c72f56c4b848f748e9dcae760c7 in term 0.
I20260929 18:53:38.026593 5305 leader_election.cc:302] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e5965c72f56c4b848f748e9dcae760c7; no voters:
I20260929 18:53:38.026731 5592 raft_consensus.cc:2781] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:53:38.026785 5592 raft_consensus.cc:482] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:53:38.026803 5592 raft_consensus.cc:3037] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.027758 5592 raft_consensus.cc:504] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:38.027897 5592 leader_election.cc:288] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [CANDIDATE]: Term 1 election: Requested vote from peers a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.028076 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb84878dd2c4a2f9154f280634e4f91" candidate_uuid: "e5965c72f56c4b848f748e9dcae760c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f5bd0bcc694c388151d8c591f9a42a"
I20260929 18:53:38.028120 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb84878dd2c4a2f9154f280634e4f91" candidate_uuid: "e5965c72f56c4b848f748e9dcae760c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
I20260929 18:53:38.028182 5233 raft_consensus.cc:3037] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.028179 5513 raft_consensus.cc:3037] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.028842 5233 raft_consensus.cc:2454] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5965c72f56c4b848f748e9dcae760c7 in term 1.
I20260929 18:53:38.028918 5513 raft_consensus.cc:2454] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5965c72f56c4b848f748e9dcae760c7 in term 1.
I20260929 18:53:38.029052 5305 leader_election.cc:302] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e5965c72f56c4b848f748e9dcae760c7; no voters:
I20260929 18:53:38.029140 5592 raft_consensus.cc:2781] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:38.029292 5592 raft_consensus.cc:686] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 1 LEADER]: Becoming Leader. State: Replica: e5965c72f56c4b848f748e9dcae760c7, State: Running, Role: LEADER
I20260929 18:53:38.029394 5592 consensus_queue.cc:237] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } }
I20260929 18:53:38.030249 5075 catalog_manager.cc:5495] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130). New cstate: current_term: 1 leader_uuid: "e5965c72f56c4b848f748e9dcae760c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } health_report { overall_health: HEALTHY } } }
I20260929 18:53:38.041635 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.039162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.29.71.130:38645)
0929 18:53:38.039219 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.039222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.039229 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:53:38.039261 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.039262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.039279 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.039364 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:53:38.039514 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.040045 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.040803 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.041011 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.041090 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.041272 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:53:38.041381 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:53:38.041490 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.041535 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.041553 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:53:38.043514 5513 raft_consensus.cc:1261] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Refusing update from remote peer e5965c72f56c4b848f748e9dcae760c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.043571 5233 raft_consensus.cc:1261] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Refusing update from remote peer e5965c72f56c4b848f748e9dcae760c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.043900 5592 consensus_queue.cc:1035] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.044021 5597 consensus_queue.cc:1035] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.046727 5601 mvcc.cc:205] Tried to move back new op lower bound from 7334740041903562752 to 7334740041848881152. Current Snapshot: MvccSnapshot[applied={T|T < 7334740041903562752}]
I20260929 18:53:38.046933 5602 mvcc.cc:205] Tried to move back new op lower bound from 7334740041903562752 to 7334740041848881152. Current Snapshot: MvccSnapshot[applied={T|T < 7334740041903562752}]
I20260929 18:53:38.048501 5076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.048605 5076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.049337 5076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Sending DeleteTablet for 3 replicas of tablet dcb84878dd2c4a2f9154f280634e4f91
I20260929 18:53:38.049696 5493 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb84878dd2c4a2f9154f280634e4f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46102
I20260929 18:53:38.049683 5213 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb84878dd2c4a2f9154f280634e4f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:35652
I20260929 18:53:38.049888 5610 tablet_replica.cc:330] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: stopping tablet replica
I20260929 18:53:38.050027 5610 raft_consensus.cc:2229] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.050012 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb84878dd2c4a2f9154f280634e4f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260929 18:53:38.050186 5610 raft_consensus.cc:2258] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.050129 5076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:53:38.050235 5611 tablet_replica.cc:330] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: stopping tablet replica
W20260929 18:53:38.050292 5076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:53:38.050316 5611 raft_consensus.cc:2229] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.050675 5611 raft_consensus.cc:2258] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.051059 5610 ts_tablet_manager.cc:1905] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.051126 5611 ts_tablet_manager.cc:1905] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.051380 5612 tablet_replica.cc:330] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: stopping tablet replica
I20260929 18:53:38.051488 5353 tablet_service.cc:1467] Processing CreateTablet for tablet 1f44ba0882b9404996121fae7c55decb (DEFAULT_TABLE table=test-table [id=e3eda8363baf4604ab26ddc322710713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.051532 5612 raft_consensus.cc:2229] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:53:38.051615 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f44ba0882b9404996121fae7c55decb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.051620 5612 raft_consensus.cc:2258] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.052053 5612 ts_tablet_manager.cc:1905] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.052950 5611 ts_tablet_manager.cc:1918] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:38.052978 5610 ts_tablet_manager.cc:1918] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:38.053017 5611 log.cc:1196] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/dcb84878dd2c4a2f9154f280634e4f91
I20260929 18:53:38.053059 5610 log.cc:1196] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/dcb84878dd2c4a2f9154f280634e4f91
I20260929 18:53:38.053097 5588 tablet_bootstrap.cc:492] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: Bootstrap starting.
I20260929 18:53:38.053292 5611 ts_tablet_manager.cc:1939] T dcb84878dd2c4a2f9154f280634e4f91 P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting consensus metadata
I20260929 18:53:38.053319 5610 ts_tablet_manager.cc:1939] T dcb84878dd2c4a2f9154f280634e4f91 P a2f5bd0bcc694c388151d8c591f9a42a: Deleting consensus metadata
I20260929 18:53:38.053503 5588 tablet_bootstrap.cc:654] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.053683 5612 ts_tablet_manager.cc:1918] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:38.053741 5612 log.cc:1196] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/dcb84878dd2c4a2f9154f280634e4f91
I20260929 18:53:38.054051 5612 ts_tablet_manager.cc:1939] T dcb84878dd2c4a2f9154f280634e4f91 P e5965c72f56c4b848f748e9dcae760c7: Deleting consensus metadata
I20260929 18:53:38.054117 5588 tablet_bootstrap.cc:492] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: No bootstrap required, opened a new log
I20260929 18:53:38.054205 5588 ts_tablet_manager.cc:1397] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:38.054211 5061 catalog_manager.cc:4841] TS a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297): tablet dcb84878dd2c4a2f9154f280634e4f91 (table test-table [id=cb8231562354415ba7ca41e900006b9f]) successfully deleted
I20260929 18:53:38.054296 5061 catalog_manager.cc:4841] TS e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629): tablet dcb84878dd2c4a2f9154f280634e4f91 (table test-table [id=cb8231562354415ba7ca41e900006b9f]) successfully deleted
I20260929 18:53:38.054368 5588 raft_consensus.cc:348] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.054440 5588 raft_consensus.cc:374] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.054468 5588 raft_consensus.cc:729] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5965c72f56c4b848f748e9dcae760c7, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.054499 5493 tablet_service.cc:1467] Processing CreateTablet for tablet 1f44ba0882b9404996121fae7c55decb (DEFAULT_TABLE table=test-table [id=e3eda8363baf4604ab26ddc322710713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.054517 5588 consensus_queue.cc:260] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.054545 5213 tablet_service.cc:1467] Processing CreateTablet for tablet 1f44ba0882b9404996121fae7c55decb (DEFAULT_TABLE table=test-table [id=e3eda8363baf4604ab26ddc322710713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.054603 5493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f44ba0882b9404996121fae7c55decb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.054601 5060 catalog_manager.cc:4841] TS e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645): tablet dcb84878dd2c4a2f9154f280634e4f91 (table test-table [id=cb8231562354415ba7ca41e900006b9f]) successfully deleted
I20260929 18:53:38.054623 5588 ts_tablet_manager.cc:1428] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:38.054641 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f44ba0882b9404996121fae7c55decb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.055759 5587 tablet_bootstrap.cc:492] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Bootstrap starting.
I20260929 18:53:38.055761 5593 tablet_bootstrap.cc:492] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: Bootstrap starting.
I20260929 18:53:38.056102 5593 tablet_bootstrap.cc:654] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.056185 5587 tablet_bootstrap.cc:654] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.056641 5593 tablet_bootstrap.cc:492] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: No bootstrap required, opened a new log
I20260929 18:53:38.056681 5593 ts_tablet_manager.cc:1397] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:38.056787 5593 raft_consensus.cc:348] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.056838 5593 raft_consensus.cc:374] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.056850 5593 raft_consensus.cc:729] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.056911 5593 consensus_queue.cc:260] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.057189 5593 ts_tablet_manager.cc:1428] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:38.057570 5587 tablet_bootstrap.cc:492] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: No bootstrap required, opened a new log
I20260929 18:53:38.057617 5587 ts_tablet_manager.cc:1397] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:38.057768 5587 raft_consensus.cc:348] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.057832 5587 raft_consensus.cc:374] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.057864 5587 raft_consensus.cc:729] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30af67ffb884326b3d2f4f3cdf2e2a1, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.057911 5587 consensus_queue.cc:260] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.057989 5587 ts_tablet_manager.cc:1428] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260929 18:53:38.066233 5560 tablet.cc:2369] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:53:38.301843 5595 raft_consensus.cc:482] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:53:38.301972 5595 raft_consensus.cc:504] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.302416 5595 leader_election.cc:288] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.305485 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.302762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:34097 (local address 127.29.71.131:34629)
0929 18:53:38.302810 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.302813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.302963 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:53:38.302994 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.302995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.303004 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.303081 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:38.303229 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.303732 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.304756 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.305119 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.305206 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.305216 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.305370 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.305410 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:53:38.305505 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.302731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:53591 (local address 127.29.71.130:38645)
0929 18:53:38.302832 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.302834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.302871 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:53:38.302975 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.302976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.302989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.303088 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:38.303298 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.303730 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.304762 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.305122 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.305206 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.305215 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.305402 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.305441 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:53:38.305531 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.302660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:34629 (local address 127.29.71.129:34097)
0929 18:53:38.302853 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.302863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.302946 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.303081 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.303082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.303085 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:38.303161 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.303163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.303819 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.303821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.304662 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.304664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.305230 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.305232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.305297 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.305300 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.305343 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.305395 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":33,"thread_start_us":94,"threads_started":1}
I20260929 18:53:38.305536 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.302654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:38645 (local address 127.29.71.129:53591)
0929 18:53:38.302802 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.302814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.302890 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.303105 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.303106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.303108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:38.303228 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.303230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.303817 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.303818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.304663 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.304665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.305219 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.305221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.305312 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.305315 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.305388 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.305431 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":41,"thread_start_us":73,"threads_started":1}
I20260929 18:53:38.305776 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f44ba0882b9404996121fae7c55decb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" is_pre_election: true
I20260929 18:53:38.305876 5513 raft_consensus.cc:2454] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.305868 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f44ba0882b9404996121fae7c55decb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7" is_pre_election: true
I20260929 18:53:38.305999 5373 raft_consensus.cc:2454] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.306077 5165 leader_election.cc:302] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e30af67ffb884326b3d2f4f3cdf2e2a1; no voters:
I20260929 18:53:38.306231 5595 raft_consensus.cc:2781] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:53:38.306279 5595 raft_consensus.cc:482] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:53:38.306294 5595 raft_consensus.cc:3037] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.306918 5595 raft_consensus.cc:504] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.307054 5595 leader_election.cc:288] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Requested vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.307171 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f44ba0882b9404996121fae7c55decb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7"
I20260929 18:53:38.307188 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f44ba0882b9404996121fae7c55decb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
I20260929 18:53:38.307251 5513 raft_consensus.cc:3037] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.307251 5373 raft_consensus.cc:3037] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.307979 5513 raft_consensus.cc:2454] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.308053 5373 raft_consensus.cc:2454] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.308173 5165 leader_election.cc:302] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e30af67ffb884326b3d2f4f3cdf2e2a1; no voters:
I20260929 18:53:38.308264 5595 raft_consensus.cc:2781] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:38.308410 5595 raft_consensus.cc:686] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 LEADER]: Becoming Leader. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Running, Role: LEADER
I20260929 18:53:38.308490 5595 consensus_queue.cc:237] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.309218 5075 catalog_manager.cc:5495] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129). New cstate: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708017, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.190@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:38.369783 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.362869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37608 (local address 127.29.71.190:43919)
0929 18:53:38.363011 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.363013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.363051 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:53:38.363119 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.363120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.363130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.363194 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:38.363626 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.364195 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.364917 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.365130 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.365189 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.367319 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:38.367320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:38.367321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:38.367326 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:38.368787 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:38.369146 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:38.369147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:38.369147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:38.369197 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:38.369365 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:38.369366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:38.369366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:38.369473 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:38.369523 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.369649 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.369717 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260929 18:53:38.370005 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.362831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43919 (local address 127.0.0.1:37608)
0929 18:53:38.362992 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.363002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.363068 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.363211 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.363211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.363515 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:38.363560 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.363562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.364267 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.364268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.364814 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.364816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.365198 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.365199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.365239 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.365772 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:38.365777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:38.367238 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:38.368858 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:38.368864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:38.368866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:38.369082 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:38.369253 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:38.369254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:38.369255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:38.369312 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:38.369526 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:38.369531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:38.369571 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.369631 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.369788 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":63,"thread_start_us":72,"threads_started":1}
I20260929 18:53:38.372012 5075 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:37608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:53:38.372181 5075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:53:38.373659 5213 tablet_service.cc:1467] Processing CreateTablet for tablet 33545488428e4506b294b2aaa417e860 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81fe47b8046642e996a536b6162f9566]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:53:38.373701 5353 tablet_service.cc:1467] Processing CreateTablet for tablet 33545488428e4506b294b2aaa417e860 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81fe47b8046642e996a536b6162f9566]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:53:38.373725 5493 tablet_service.cc:1467] Processing CreateTablet for tablet 33545488428e4506b294b2aaa417e860 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81fe47b8046642e996a536b6162f9566]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:53:38.373837 5493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33545488428e4506b294b2aaa417e860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.373852 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33545488428e4506b294b2aaa417e860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.373823 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33545488428e4506b294b2aaa417e860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.375145 5588 tablet_bootstrap.cc:492] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: Bootstrap starting.
I20260929 18:53:38.375499 5588 tablet_bootstrap.cc:654] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.375761 5587 tablet_bootstrap.cc:492] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: Bootstrap starting.
I20260929 18:53:38.376094 5588 tablet_bootstrap.cc:492] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: No bootstrap required, opened a new log
I20260929 18:53:38.376151 5588 ts_tablet_manager.cc:1397] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:38.376178 5587 tablet_bootstrap.cc:654] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.376267 5588 raft_consensus.cc:348] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.376354 5588 raft_consensus.cc:374] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.376379 5588 raft_consensus.cc:729] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5965c72f56c4b848f748e9dcae760c7, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.376428 5588 consensus_queue.cc:260] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.376499 5593 tablet_bootstrap.cc:492] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: Bootstrap starting.
I20260929 18:53:38.376537 5588 ts_tablet_manager.cc:1428] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:38.376582 5592 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.376686 5592 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:53:38.376911 5593 tablet_bootstrap.cc:654] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.377512 5587 tablet_bootstrap.cc:492] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: No bootstrap required, opened a new log
I20260929 18:53:38.377552 5587 ts_tablet_manager.cc:1397] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:38.377719 5587 raft_consensus.cc:348] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.377775 5587 raft_consensus.cc:374] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.377789 5587 raft_consensus.cc:729] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30af67ffb884326b3d2f4f3cdf2e2a1, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.377844 5587 consensus_queue.cc:260] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.377949 5587 ts_tablet_manager.cc:1428] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:38.378008 5591 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.378125 5591 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:53:38.378171 5593 tablet_bootstrap.cc:492] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: No bootstrap required, opened a new log
I20260929 18:53:38.378214 5593 ts_tablet_manager.cc:1397] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:38.378362 5593 raft_consensus.cc:348] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.378443 5593 raft_consensus.cc:374] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.378463 5593 raft_consensus.cc:729] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.378508 5593 consensus_queue.cc:260] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.378604 5593 ts_tablet_manager.cc:1428] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:38.378650 5595 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.378744 5595 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:53:38.548712 5595 raft_consensus.cc:482] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:53:38.548810 5595 raft_consensus.cc:504] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.548964 5595 leader_election.cc:288] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.549216 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33545488428e4506b294b2aaa417e860" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7" is_pre_election: true
I20260929 18:53:38.549337 5373 raft_consensus.cc:2454] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.549247 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33545488428e4506b294b2aaa417e860" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" is_pre_election: true
I20260929 18:53:38.549387 5513 raft_consensus.cc:2454] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.549549 5165 leader_election.cc:302] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e30af67ffb884326b3d2f4f3cdf2e2a1; no voters:
I20260929 18:53:38.549660 5595 raft_consensus.cc:2781] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:53:38.549710 5595 raft_consensus.cc:482] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:53:38.549739 5595 raft_consensus.cc:3037] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.550542 5595 raft_consensus.cc:504] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.550698 5595 leader_election.cc:288] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Requested vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.550913 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33545488428e4506b294b2aaa417e860" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7"
I20260929 18:53:38.550900 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33545488428e4506b294b2aaa417e860" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
I20260929 18:53:38.550979 5513 raft_consensus.cc:3037] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.551002 5373 raft_consensus.cc:3037] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.551584 5373 raft_consensus.cc:2454] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.551784 5164 leader_election.cc:302] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e5965c72f56c4b848f748e9dcae760c7; no voters:
I20260929 18:53:38.551954 5595 raft_consensus.cc:2781] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:38.552028 5595 raft_consensus.cc:686] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 LEADER]: Becoming Leader. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Running, Role: LEADER
I20260929 18:53:38.552083 5595 consensus_queue.cc:237] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.552353 5513 raft_consensus.cc:2454] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.552409 5618 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: TxnStatusTablet state changed. Reason: New leader a2f5bd0bcc694c388151d8c591f9a42a. Latest consensus state: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.552500 5618 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:53:38.552765 5631 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:53:38.552817 5075 catalog_manager.cc:5495] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129). New cstate: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } health_report { overall_health: UNKNOWN } } }
I20260929 18:53:38.552937 5631 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:53:38.559918 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.556942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.29.71.129:36297)
0929 18:53:38.557042 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.557044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.557052 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:38.557084 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.557085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.557116 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.557206 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:53:38.557377 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.557915 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.558808 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.559182 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.559297 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.559545 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:53:38.559649 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:53:38.559735 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.559788 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.559810 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":45,"threads_started":1}
I20260929 18:53:38.559906 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.556837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36297 (local address 127.0.0.1:35664)
0929 18:53:38.556917 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.556930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.556986 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.557208 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.557209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.557210 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:53:38.557262 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.557264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.558002 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.558003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.558707 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.558709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.559300 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.559301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.559371 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.559394 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:53:38.559718 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:53:38.559727 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.559783 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.559838 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 18:53:38.561108 5513 raft_consensus.cc:1261] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.561225 5373 raft_consensus.cc:1261] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.561524 5595 consensus_queue.cc:1035] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.561676 5618 consensus_queue.cc:1035] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.562254 5591 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: TxnStatusTablet state changed. Reason: New leader a2f5bd0bcc694c388151d8c591f9a42a. Latest consensus state: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.562345 5591 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:53:38.562333 5592 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: TxnStatusTablet state changed. Reason: New leader a2f5bd0bcc694c388151d8c591f9a42a. Latest consensus state: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.562413 5592 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:53:38.563288 5595 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.563370 5595 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:53:38.563416 5595 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.563392 5592 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.563455 5592 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P e5965c72f56c4b848f748e9dcae760c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:53:38.563453 5595 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P a2f5bd0bcc694c388151d8c591f9a42a: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:53:38.563957 5591 tablet_replica.cc:439] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } } }
I20260929 18:53:38.564050 5591 tablet_replica.cc:442] T 33545488428e4506b294b2aaa417e860 P e30af67ffb884326b3d2f4f3cdf2e2a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:53:38.564929 5638 mvcc.cc:205] Tried to move back new op lower bound from 7334740044024487936 to 7334740043989979136. Current Snapshot: MvccSnapshot[applied={T|T < 7334740044024487936}]
I20260929 18:53:38.567781 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.565301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.29.71.129:36297)
0929 18:53:38.565340 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.565342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.565396 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:53:38.565473 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.565474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.565485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.565540 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:53:38.565663 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.566211 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.567092 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.567317 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.567402 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.567546 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:53:38.567579 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:53:38.567620 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.567684 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.567703 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:53:38.572878 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.570152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:38231 (local address 127.29.71.129:36297)
0929 18:53:38.570211 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.570214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.570324 (+ 110us) server_negotiation.cc:407] Connection header received
0929 18:53:38.570388 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.570388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.570404 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.570449 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:53:38.570606 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.571125 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.572247 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.572418 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.572462 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.572660 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:53:38.572683 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:53:38.572717 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.572796 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.572811 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:53:38.572975 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.570075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36297 (local address 127.29.71.129:38231)
0929 18:53:38.570259 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.570271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.570333 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.570470 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.570470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.570472 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:53:38.570547 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.570548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.571200 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.571201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.572161 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.572163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.572497 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.572498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.572561 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.572577 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:53:38.572739 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:53:38.572741 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.572796 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.572841 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":137,"threads_started":1}
I20260929 18:53:38.575031 5373 raft_consensus.cc:1261] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.575035 5513 raft_consensus.cc:1261] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.575241 5648 consensus_queue.cc:1035] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.575352 5618 consensus_queue.cc:1035] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.587379 5075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.587493 5075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.588155 5075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Sending DeleteTablet for 3 replicas of tablet 1f44ba0882b9404996121fae7c55decb
I20260929 18:53:38.588398 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f44ba0882b9404996121fae7c55decb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260929 18:53:38.588483 5493 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f44ba0882b9404996121fae7c55decb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46102
I20260929 18:53:38.588650 5653 tablet_replica.cc:330] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: stopping tablet replica
I20260929 18:53:38.588726 5653 raft_consensus.cc:2229] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.588794 5653 raft_consensus.cc:2258] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.588769 5654 tablet_replica.cc:330] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: stopping tablet replica
I20260929 18:53:38.588896 5213 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f44ba0882b9404996121fae7c55decb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:35652
I20260929 18:53:38.588871 5654 raft_consensus.cc:2229] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.589025 5654 raft_consensus.cc:2258] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.589077 5653 ts_tablet_manager.cc:1905] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.589273 5654 ts_tablet_manager.cc:1905] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.591212 5654 ts_tablet_manager.cc:1918] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:53:38.591321 5654 log.cc:1196] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/1f44ba0882b9404996121fae7c55decb
I20260929 18:53:38.591598 5654 ts_tablet_manager.cc:1939] T 1f44ba0882b9404996121fae7c55decb P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting consensus metadata
I20260929 18:53:38.591952 5061 catalog_manager.cc:4841] TS e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629): tablet 1f44ba0882b9404996121fae7c55decb (table test-table [id=e3eda8363baf4604ab26ddc322710713]) successfully deleted
I20260929 18:53:38.592199 5653 ts_tablet_manager.cc:1918] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:53:38.592273 5653 log.cc:1196] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/1f44ba0882b9404996121fae7c55decb
I20260929 18:53:38.592536 5653 ts_tablet_manager.cc:1939] T 1f44ba0882b9404996121fae7c55decb P e5965c72f56c4b848f748e9dcae760c7: Deleting consensus metadata
I20260929 18:53:38.592882 5060 catalog_manager.cc:4841] TS e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645): tablet 1f44ba0882b9404996121fae7c55decb (table test-table [id=e3eda8363baf4604ab26ddc322710713]) successfully deleted
I20260929 18:53:38.594270 5655 tablet_replica.cc:330] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: stopping tablet replica
I20260929 18:53:38.594352 5655 raft_consensus.cc:2229] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:53:38.594481 5655 raft_consensus.cc:2258] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.594728 5655 ts_tablet_manager.cc:1905] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.596741 5655 ts_tablet_manager.cc:1918] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:53:38.596817 5655 log.cc:1196] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/1f44ba0882b9404996121fae7c55decb
I20260929 18:53:38.597067 5655 ts_tablet_manager.cc:1939] T 1f44ba0882b9404996121fae7c55decb P a2f5bd0bcc694c388151d8c591f9a42a: Deleting consensus metadata
I20260929 18:53:38.597380 5061 catalog_manager.cc:4841] TS a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297): tablet 1f44ba0882b9404996121fae7c55decb (table test-table [id=e3eda8363baf4604ab26ddc322710713]) successfully deleted
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:53:38.614001 5626 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:38.607670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.29.71.190:43919)
0929 18:53:38.607724 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.607726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.607822 (+ 96us) server_negotiation.cc:407] Connection header received
0929 18:53:38.607899 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.607900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.607912 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.607970 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:38.608516 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.609343 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.610110 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.610454 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.610531 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.613926 (+ 3395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37610: BlockingRecv error: recv got EOF from 127.0.0.1:37610 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM
I20260929 18:53:38.623423 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.615196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.29.71.190:43919)
0929 18:53:38.615242 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.615245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.615273 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:53:38.615302 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.615303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.615314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.615395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:38.615988 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.616537 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.617445 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.617659 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.617739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.620364 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:38.620366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:38.620367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:38.620379 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:38.622046 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:38.622518 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:38.622520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:38.622520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:38.622582 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:38.622803 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:38.622804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:38.622805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:38.622941 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:38.623055 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.623203 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.623297 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:53:38.625228 5075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:53:38.625425 5075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:53:38.626950 5213 tablet_service.cc:1467] Processing CreateTablet for tablet bc2dacf2fd694f9b92564c4489861cfb (DEFAULT_TABLE table=test-table [id=c9a26d44b6a14be9b6829b5a5d16425a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.626993 5493 tablet_service.cc:1467] Processing CreateTablet for tablet bc2dacf2fd694f9b92564c4489861cfb (DEFAULT_TABLE table=test-table [id=c9a26d44b6a14be9b6829b5a5d16425a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.627134 5493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2dacf2fd694f9b92564c4489861cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.627130 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2dacf2fd694f9b92564c4489861cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.628161 5593 tablet_bootstrap.cc:492] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: Bootstrap starting.
I20260929 18:53:38.628228 5353 tablet_service.cc:1467] Processing CreateTablet for tablet bc2dacf2fd694f9b92564c4489861cfb (DEFAULT_TABLE table=test-table [id=c9a26d44b6a14be9b6829b5a5d16425a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.628367 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2dacf2fd694f9b92564c4489861cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.628444 5587 tablet_bootstrap.cc:492] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: Bootstrap starting.
I20260929 18:53:38.628593 5593 tablet_bootstrap.cc:654] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.628835 5587 tablet_bootstrap.cc:654] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.629531 5587 tablet_bootstrap.cc:492] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: No bootstrap required, opened a new log
I20260929 18:53:38.629585 5587 ts_tablet_manager.cc:1397] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:38.629727 5587 raft_consensus.cc:348] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.629766 5588 tablet_bootstrap.cc:492] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: Bootstrap starting.
I20260929 18:53:38.629797 5587 raft_consensus.cc:374] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.629817 5587 raft_consensus.cc:729] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30af67ffb884326b3d2f4f3cdf2e2a1, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.629855 5587 consensus_queue.cc:260] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.629938 5587 ts_tablet_manager.cc:1428] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:38.630198 5588 tablet_bootstrap.cc:654] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.630364 5593 tablet_bootstrap.cc:492] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: No bootstrap required, opened a new log
I20260929 18:53:38.630419 5593 ts_tablet_manager.cc:1397] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:38.630528 5593 raft_consensus.cc:348] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.630612 5593 raft_consensus.cc:374] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.630637 5593 raft_consensus.cc:729] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.630692 5593 consensus_queue.cc:260] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.630791 5593 ts_tablet_manager.cc:1428] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:38.631388 5588 tablet_bootstrap.cc:492] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: No bootstrap required, opened a new log
I20260929 18:53:38.631443 5588 ts_tablet_manager.cc:1397] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:53:38.631544 5588 raft_consensus.cc:348] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.631614 5588 raft_consensus.cc:374] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.631633 5588 raft_consensus.cc:729] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5965c72f56c4b848f748e9dcae760c7, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.631683 5588 consensus_queue.cc:260] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.631767 5588 ts_tablet_manager.cc:1428] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:38.650305 5591 raft_consensus.cc:482] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:53:38.650419 5591 raft_consensus.cc:504] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.650753 5591 leader_election.cc:288] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297), e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645)
I20260929 18:53:38.652803 5651 raft_consensus.cc:482] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:53:38.652879 5651 raft_consensus.cc:504] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.653038 5651 leader_election.cc:288] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.653347 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2dacf2fd694f9b92564c4489861cfb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" is_pre_election: true
I20260929 18:53:38.653432 5513 raft_consensus.cc:2454] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.653620 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2dacf2fd694f9b92564c4489861cfb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7" is_pre_election: true
I20260929 18:53:38.653700 5373 raft_consensus.cc:2454] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.653949 5165 leader_election.cc:302] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e30af67ffb884326b3d2f4f3cdf2e2a1; no voters:
I20260929 18:53:38.654268 5651 raft_consensus.cc:2781] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:53:38.654313 5651 raft_consensus.cc:482] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:53:38.654335 5651 raft_consensus.cc:3037] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.655262 5651 raft_consensus.cc:504] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.655385 5651 leader_election.cc:288] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Requested vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.655375 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.651363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:51607 (local address 127.29.71.130:38645)
0929 18:53:38.651402 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.651404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.651794 (+ 390us) server_negotiation.cc:407] Connection header received
0929 18:53:38.651900 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.651901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.651910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.651939 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:38.652184 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.652871 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.654428 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.654964 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.655016 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.655038 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.655270 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.655308 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:53:38.655644 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.651080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:38645 (local address 127.29.71.131:51607)
0929 18:53:38.651733 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.651744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.651823 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.652001 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.652002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.652004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:38.652099 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.652100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.652978 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.652979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.654293 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.654295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.655112 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.655113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.655187 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.655190 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.655517 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.655562 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":106,"threads_started":1}
I20260929 18:53:38.655644 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2dacf2fd694f9b92564c4489861cfb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7"
I20260929 18:53:38.655714 5373 raft_consensus.cc:3037] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.655927 5372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2dacf2fd694f9b92564c4489861cfb" candidate_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7" is_pre_election: true
I20260929 18:53:38.655972 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2dacf2fd694f9b92564c4489861cfb" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
I20260929 18:53:38.656038 5513 raft_consensus.cc:3037] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.656563 5373 raft_consensus.cc:2454] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.656737 5164 leader_election.cc:302] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e5965c72f56c4b848f748e9dcae760c7; no voters:
I20260929 18:53:38.656899 5513 raft_consensus.cc:2454] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.656908 5651 raft_consensus.cc:2781] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:38.656958 5651 raft_consensus.cc:686] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 LEADER]: Becoming Leader. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Running, Role: LEADER
I20260929 18:53:38.657033 5651 consensus_queue.cc:237] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.657581 5075 catalog_manager.cc:5495] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129). New cstate: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } health_report { overall_health: UNKNOWN } } }
I20260929 18:53:38.658717 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.650995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36297 (local address 127.29.71.131:51877)
0929 18:53:38.651271 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.651284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:38.651341 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:38.651638 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:38.651639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:38.651641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:53:38.651743 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.651745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.656006 (+ 4261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.656007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.657614 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:38.657617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.658455 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.658457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:38.658541 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.658545 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:53:38.658609 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:53:38.658658 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":102,"threads_started":1}
I20260929 18:53:38.658732 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.651262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:51877 (local address 127.29.71.129:36297)
0929 18:53:38.651307 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.651309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.651324 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:53:38.651513 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.651514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.651526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.651560 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:53:38.651826 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.652671 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.657708 (+ 5037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.658336 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.658407 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.658418 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.658633 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.658675 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:53:38.659029 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2dacf2fd694f9b92564c4489861cfb" candidate_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" is_pre_election: true
I20260929 18:53:38.659273 5445 leader_election.cc:302] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e30af67ffb884326b3d2f4f3cdf2e2a1; no voters: a2f5bd0bcc694c388151d8c591f9a42a, e5965c72f56c4b848f748e9dcae760c7
I20260929 18:53:38.659480 5591 raft_consensus.cc:2726] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:53:38.670137 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.667622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.29.71.129:36297)
0929 18:53:38.667664 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:38.667667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:38.667680 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:53:38.667765 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:38.667766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:38.667779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:38.667848 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:53:38.667954 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.668643 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.669439 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:38.669700 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:38.669774 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:38.669898 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:53:38.669930 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:53:38.669982 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:38.670041 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:53:38.670062 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:53:38.670980 5513 raft_consensus.cc:1261] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.671068 5373 raft_consensus.cc:1261] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.671228 5651 consensus_queue.cc:1035] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.671375 5618 consensus_queue.cc:1035] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.674227 5074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.674335 5074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.674995 5074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Sending DeleteTablet for 3 replicas of tablet bc2dacf2fd694f9b92564c4489861cfb
I20260929 18:53:38.675235 5493 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2dacf2fd694f9b92564c4489861cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46102
I20260929 18:53:38.675300 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2dacf2fd694f9b92564c4489861cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260929 18:53:38.675344 5492 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2dacf2fd694f9b92564c4489861cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46102
I20260929 18:53:38.675405 5654 tablet_replica.cc:330] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: stopping tablet replica
I20260929 18:53:38.675400 5653 tablet_replica.cc:330] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: stopping tablet replica
I20260929 18:53:38.675446 5654 raft_consensus.cc:2229] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.675452 5653 raft_consensus.cc:2229] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.675485 5653 raft_consensus.cc:2258] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.675485 5654 raft_consensus.cc:2258] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.675597 5213 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2dacf2fd694f9b92564c4489861cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:35652
I20260929 18:53:38.675665 5655 tablet_replica.cc:330] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: stopping tablet replica
I20260929 18:53:38.675707 5655 raft_consensus.cc:2229] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:53:38.675756 5655 raft_consensus.cc:2258] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.675840 5653 ts_tablet_manager.cc:1905] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.676097 5655 ts_tablet_manager.cc:1905] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.677554 5655 ts_tablet_manager.cc:1918] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:38.677579 5653 ts_tablet_manager.cc:1918] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:38.677609 5655 log.cc:1196] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/bc2dacf2fd694f9b92564c4489861cfb
I20260929 18:53:38.677630 5653 log.cc:1196] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/bc2dacf2fd694f9b92564c4489861cfb
I20260929 18:53:38.677852 5655 ts_tablet_manager.cc:1939] T bc2dacf2fd694f9b92564c4489861cfb P a2f5bd0bcc694c388151d8c591f9a42a: Deleting consensus metadata
I20260929 18:53:38.677852 5653 ts_tablet_manager.cc:1939] T bc2dacf2fd694f9b92564c4489861cfb P e5965c72f56c4b848f748e9dcae760c7: Deleting consensus metadata
I20260929 18:53:38.678148 5654 ts_tablet_manager.cc:1905] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.678095 5074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:53:38.678325 5074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:53:38.679571 5060 catalog_manager.cc:4841] TS e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645): tablet bc2dacf2fd694f9b92564c4489861cfb (table test-table [id=c9a26d44b6a14be9b6829b5a5d16425a]) successfully deleted
I20260929 18:53:38.679612 5492 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5e2ea069744d488d6c8022ecd87080 (DEFAULT_TABLE table=test-table [id=b313444e978e452ab8e4ede6d0c8a7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.679741 5492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5e2ea069744d488d6c8022ecd87080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.679749 5353 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5e2ea069744d488d6c8022ecd87080 (DEFAULT_TABLE table=test-table [id=b313444e978e452ab8e4ede6d0c8a7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.679786 5213 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5e2ea069744d488d6c8022ecd87080 (DEFAULT_TABLE table=test-table [id=b313444e978e452ab8e4ede6d0c8a7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:53:38.679836 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5e2ea069744d488d6c8022ecd87080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.679883 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5e2ea069744d488d6c8022ecd87080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:38.681073 5593 tablet_bootstrap.cc:492] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: Bootstrap starting.
I20260929 18:53:38.681097 5588 tablet_bootstrap.cc:492] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: Bootstrap starting.
I20260929 18:53:38.681427 5588 tablet_bootstrap.cc:654] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.681500 5593 tablet_bootstrap.cc:654] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.681828 5587 tablet_bootstrap.cc:492] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: Bootstrap starting.
I20260929 18:53:38.682116 5588 tablet_bootstrap.cc:492] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: No bootstrap required, opened a new log
I20260929 18:53:38.682150 5588 ts_tablet_manager.cc:1397] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:38.682263 5587 tablet_bootstrap.cc:654] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:38.682322 5061 catalog_manager.cc:4841] TS a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297): tablet bc2dacf2fd694f9b92564c4489861cfb (table test-table [id=c9a26d44b6a14be9b6829b5a5d16425a]) successfully deleted
I20260929 18:53:38.682317 5588 raft_consensus.cc:348] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.682408 5588 raft_consensus.cc:374] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.682437 5588 raft_consensus.cc:729] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5965c72f56c4b848f748e9dcae760c7, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.679626 5654 ts_tablet_manager.cc:1918] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:53:38.682489 5588 consensus_queue.cc:260] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.682555 5654 log.cc:1196] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/bc2dacf2fd694f9b92564c4489861cfb
I20260929 18:53:38.682600 5588 ts_tablet_manager.cc:1428] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:38.682822 5654 ts_tablet_manager.cc:1939] T bc2dacf2fd694f9b92564c4489861cfb P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting consensus metadata
I20260929 18:53:38.682883 5587 tablet_bootstrap.cc:492] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: No bootstrap required, opened a new log
I20260929 18:53:38.682924 5587 ts_tablet_manager.cc:1397] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:38.683068 5587 raft_consensus.cc:348] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.683136 5587 raft_consensus.cc:374] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.683156 5587 raft_consensus.cc:729] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30af67ffb884326b3d2f4f3cdf2e2a1, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.683193 5587 consensus_queue.cc:260] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.683274 5587 ts_tablet_manager.cc:1428] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:38.683265 5061 catalog_manager.cc:4841] TS e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629): tablet bc2dacf2fd694f9b92564c4489861cfb (table test-table [id=c9a26d44b6a14be9b6829b5a5d16425a]) successfully deleted
W20260929 18:53:38.683404 5061 catalog_manager.cc:4808] TS e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629): delete failed for tablet bc2dacf2fd694f9b92564c4489861cfb because the tablet was not found. No further retry: Not found: Tablet not found: bc2dacf2fd694f9b92564c4489861cfb
I20260929 18:53:38.683895 5593 tablet_bootstrap.cc:492] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: No bootstrap required, opened a new log
I20260929 18:53:38.683951 5593 ts_tablet_manager.cc:1397] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:53:38.684109 5593 raft_consensus.cc:348] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.684177 5593 raft_consensus.cc:374] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:38.684206 5593 raft_consensus.cc:729] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Initialized, Role: FOLLOWER
I20260929 18:53:38.684253 5593 consensus_queue.cc:260] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.684332 5593 ts_tablet_manager.cc:1428] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:38.752701 5648 raft_consensus.cc:482] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:53:38.752806 5648 raft_consensus.cc:504] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.753010 5648 leader_election.cc:288] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.753252 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5e2ea069744d488d6c8022ecd87080" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7" is_pre_election: true
I20260929 18:53:38.753252 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5e2ea069744d488d6c8022ecd87080" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" is_pre_election: true
I20260929 18:53:38.753355 5373 raft_consensus.cc:2454] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.753372 5513 raft_consensus.cc:2454] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 0.
I20260929 18:53:38.753527 5165 leader_election.cc:302] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e30af67ffb884326b3d2f4f3cdf2e2a1; no voters:
I20260929 18:53:38.753633 5648 raft_consensus.cc:2781] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:53:38.753688 5648 raft_consensus.cc:482] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:53:38.753714 5648 raft_consensus.cc:3037] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.754616 5648 raft_consensus.cc:504] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.754766 5648 leader_election.cc:288] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Requested vote from peers e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645), e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629)
I20260929 18:53:38.754895 5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5e2ea069744d488d6c8022ecd87080" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5965c72f56c4b848f748e9dcae760c7"
I20260929 18:53:38.754981 5373 raft_consensus.cc:3037] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.754971 5513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5e2ea069744d488d6c8022ecd87080" candidate_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1"
I20260929 18:53:38.755040 5513 raft_consensus.cc:3037] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:38.755793 5513 raft_consensus.cc:2454] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.755802 5373 raft_consensus.cc:2454] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f5bd0bcc694c388151d8c591f9a42a in term 1.
I20260929 18:53:38.755964 5165 leader_election.cc:302] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f5bd0bcc694c388151d8c591f9a42a, e30af67ffb884326b3d2f4f3cdf2e2a1; no voters:
I20260929 18:53:38.756047 5648 raft_consensus.cc:2781] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:38.756106 5648 raft_consensus.cc:686] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 LEADER]: Becoming Leader. State: Replica: a2f5bd0bcc694c388151d8c591f9a42a, State: Running, Role: LEADER
I20260929 18:53:38.756163 5648 consensus_queue.cc:237] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } }
I20260929 18:53:38.756877 5074 catalog_manager.cc:5495] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129). New cstate: current_term: 1 leader_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f5bd0bcc694c388151d8c591f9a42a" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 } health_report { overall_health: UNKNOWN } } }
I20260929 18:53:38.781198 5512 raft_consensus.cc:1261] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.781225 5373 raft_consensus.cc:1261] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Refusing update from remote peer a2f5bd0bcc694c388151d8c591f9a42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:53:38.781412 5648 consensus_queue.cc:1035] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30af67ffb884326b3d2f4f3cdf2e2a1" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.781538 5595 consensus_queue.cc:1035] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5965c72f56c4b848f748e9dcae760c7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:53:38.791549 5074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.791641 5074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:53:38.792285 5074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeb2995c43bb43c0af5549e6317b368c: Sending DeleteTablet for 3 replicas of tablet 9a5e2ea069744d488d6c8022ecd87080
I20260929 18:53:38.792505 5213 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5e2ea069744d488d6c8022ecd87080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:35652
I20260929 18:53:38.792582 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5e2ea069744d488d6c8022ecd87080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260929 18:53:38.792582 5492 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5e2ea069744d488d6c8022ecd87080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:53:38 UTC) from {username='oryx', principal='oryx/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46102
I20260929 18:53:38.792611 5655 tablet_replica.cc:330] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: stopping tablet replica
I20260929 18:53:38.792659 5655 raft_consensus.cc:2229] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:53:38.792670 5654 tablet_replica.cc:330] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: stopping tablet replica
I20260929 18:53:38.792670 5653 tablet_replica.cc:330] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: stopping tablet replica
I20260929 18:53:38.792713 5655 raft_consensus.cc:2258] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.792716 5654 raft_consensus.cc:2229] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.792716 5653 raft_consensus.cc:2229] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:53:38.792749 5653 raft_consensus.cc:2258] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.792749 5654 raft_consensus.cc:2258] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:53:38.793079 5653 ts_tablet_manager.cc:1905] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.793118 5655 ts_tablet_manager.cc:1905] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.794647 5653 ts_tablet_manager.cc:1918] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:53:38.794677 5655 ts_tablet_manager.cc:1918] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:53:38.794708 5653 log.cc:1196] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/9a5e2ea069744d488d6c8022ecd87080
I20260929 18:53:38.794727 5655 log.cc:1196] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/9a5e2ea069744d488d6c8022ecd87080
I20260929 18:53:38.794874 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 5148
I20260929 18:53:38.794941 5655 ts_tablet_manager.cc:1939] T 9a5e2ea069744d488d6c8022ecd87080 P a2f5bd0bcc694c388151d8c591f9a42a: Deleting consensus metadata
I20260929 18:53:38.794927 5653 ts_tablet_manager.cc:1939] T 9a5e2ea069744d488d6c8022ecd87080 P e5965c72f56c4b848f748e9dcae760c7: Deleting consensus metadata
I20260929 18:53:38.795838 5654 ts_tablet_manager.cc:1905] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:53:38.797591 5654 ts_tablet_manager.cc:1918] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:53:38.797645 5654 log.cc:1196] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/9a5e2ea069744d488d6c8022ecd87080
I20260929 18:53:38.797853 5654 ts_tablet_manager.cc:1939] T 9a5e2ea069744d488d6c8022ecd87080 P e30af67ffb884326b3d2f4f3cdf2e2a1: Deleting consensus metadata
I20260929 18:53:38.798295 5061 catalog_manager.cc:4841] TS e30af67ffb884326b3d2f4f3cdf2e2a1 (127.29.71.131:34629): tablet 9a5e2ea069744d488d6c8022ecd87080 (table test-table [id=b313444e978e452ab8e4ede6d0c8a7e5]) successfully deleted
I20260929 18:53:38.798619 5060 catalog_manager.cc:4841] TS e5965c72f56c4b848f748e9dcae760c7 (127.29.71.130:38645): tablet 9a5e2ea069744d488d6c8022ecd87080 (table test-table [id=b313444e978e452ab8e4ede6d0c8a7e5]) successfully deleted
W20260929 18:53:38.803138 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:36297 (error 108)
I20260929 18:53:38.803385 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 5288
I20260929 18:53:38.804234 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:38.804019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36297 (local address 127.0.0.1:35676)
0929 18:53:38.804128 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:38.804180 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
W20260929 18:53:38.804370 5061 catalog_manager.cc:4584] TS a2f5bd0bcc694c388151d8c591f9a42a (127.29.71.129:36297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a5e2ea069744d488d6c8022ecd87080: Network error: Client connection negotiation failed: client connection to 127.29.71.129:36297: connect: Connection refused (error 111)
I20260929 18:53:38.810025 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 5428
I20260929 18:53:38.815148 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 5044
2026-09-29T18:53:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2822 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5687](info): set up 2 sockets
Sep 29 18:53:38 dist-test-slave-q2kh krb5kdc[5687](info): commencing operation
krb5kdc: starting...
W20260929 18:53:39.962687 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:53:40 dist-test-slave-q2kh krb5kdc[5687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:53:40.083288 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34419
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:34419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:40.178648 5700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:40.178848 5700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:40.178869 5700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:40.179931 5700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:40.179965 5700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:40.179978 5700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:40.179991 5700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:40.180003 5700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:40.181365 5700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:34419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:40.181551 5700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:40.181720 5700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:40.186450 5705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:40.186492 5708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:40.186602 5706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:40.186646 5700 server_base.cc:1034] running on GCE node
I20260929 18:53:40.186949 5700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:40.187206 5700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:40.188341 5700 hybrid_clock.cc:648] HybridClock initialized: now 1790708020188331 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T18:53:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1374 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:53:40 dist-test-slave-q2kh krb5kdc[5713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:53:40 dist-test-slave-q2kh krb5kdc[5713](info): set up 2 sockets
Sep 29 18:53:40 dist-test-slave-q2kh krb5kdc[5713](info): commencing operation
krb5kdc: starting...
W20260929 18:53:41.230605 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:53:41.349081 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42219
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:41.427804 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:41.428014 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:41.428036 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:41.429100 5726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:41.429131 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:41.429145 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:41.429158 5726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:41.429170 5726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:41.430562 5726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:41.430752 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:41.430913 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:53:41.435643 5726 server_base.cc:1034] running on GCE node
W20260929 18:53:41.435550 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:41.435562 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:41.435550 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:41.435976 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:41.436259 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:41.437405 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790708021437395 us; error 34 us; skew 500 ppm
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:41.443776 5726 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:53:41.444262 5726 webserver.cc:460] Webserver started at http://127.29.71.190:45115/ using document root <none> and password file <none>
I20260929 18:53:41.444505 5726 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:41.444550 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:41.444700 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:41.445750 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "887b5068543c4d4ab3b85daf020029de"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "2fc9ea3bbd8271a9aabe64efc80b302f"
server_key_iv: "490b841aac0d1d75ff7b5c8c43b8a883"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.446110 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "887b5068543c4d4ab3b85daf020029de"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "2fc9ea3bbd8271a9aabe64efc80b302f"
server_key_iv: "490b841aac0d1d75ff7b5c8c43b8a883"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.447558 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:41.448364 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.448546 5726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:41.448619 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "887b5068543c4d4ab3b85daf020029de"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "2fc9ea3bbd8271a9aabe64efc80b302f"
server_key_iv: "490b841aac0d1d75ff7b5c8c43b8a883"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.448712 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:41.467968 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:41.469841 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:41.469995 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:41.474105 5726 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:42219
I20260929 18:53:41.474145 5793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:42219 every 8 connection(s)
I20260929 18:53:41.474558 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:41.475397 5794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:41.477576 5794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Bootstrap starting.
I20260929 18:53:41.478111 5794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:41.478439 5794 log.cc:824] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:41.479142 5794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: No bootstrap required, opened a new log
I20260929 18:53:41.480640 5794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:41.480777 5794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:41.480799 5794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887b5068543c4d4ab3b85daf020029de, State: Initialized, Role: FOLLOWER
I20260929 18:53:41.480937 5794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:41.481027 5794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:41.481051 5794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:41.481079 5794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:41.481547 5794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:41.481632 5794 leader_election.cc:302] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 887b5068543c4d4ab3b85daf020029de; no voters:
I20260929 18:53:41.481781 5794 leader_election.cc:288] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:41.481828 5797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:41.482076 5794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:41.482147 5797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 LEADER]: Becoming Leader. State: Replica: 887b5068543c4d4ab3b85daf020029de, State: Running, Role: LEADER
I20260929 18:53:41.482240 5797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:41.482721 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "887b5068543c4d4ab3b85daf020029de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } } }
I20260929 18:53:41.482733 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 887b5068543c4d4ab3b85daf020029de. Latest consensus state: current_term: 1 leader_uuid: "887b5068543c4d4ab3b85daf020029de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } } }
I20260929 18:53:41.482831 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:41.482853 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:41.483239 5807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:41.483763 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 5726
I20260929 18:53:41.483883 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:41.484115 29982 external_mini_cluster.cc:1428] Setting key 05e3c01197a85b8380944ec5e2211a05
I20260929 18:53:41.483855 5807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:41.485822 5807 catalog_manager.cc:1347] Generated new cluster ID: 685c8b7a4b8144509b404bbc0db25d8d
I20260929 18:53:41.485874 5807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:53:41.495903 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:41.484596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.29.71.190:42219)
0929 18:53:41.484870 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:41.484878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:41.484901 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:53:41.485042 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:41.485048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:41.485104 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:41.485239 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:41.486013 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.487463 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.488515 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.489141 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.489257 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:41.492170 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:41.492174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:41.492174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:41.492187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:41.494187 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.494598 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.494601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.494602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.494723 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.494948 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.494949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.494949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.495261 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:41.495400 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:41.495537 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 18:53:41.495667 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260929 18:53:41.504731 5807 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:41.505270 5807 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:41.515432 5807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Generated new TSK 0
I20260929 18:53:41.515659 5807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:53:41.546639 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42219
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:41.626906 5820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:41.627122 5820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:41.627147 5820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:41.628238 5820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:41.628273 5820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:41.628297 5820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:41.629629 5820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:41.629804 5820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:41.629971 5820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:41.635056 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:41.635036 5825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:41.635118 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:41.635223 5820 server_base.cc:1034] running on GCE node
I20260929 18:53:41.635514 5820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:41.635766 5820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:41.636926 5820 hybrid_clock.cc:648] HybridClock initialized: now 1790708021636944 us; error 60 us; skew 500 ppm
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:41.639456 5820 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:53:41.639896 5820 webserver.cc:460] Webserver started at http://127.29.71.129:36565/ using document root <none> and password file <none>
I20260929 18:53:41.640123 5820 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:41.640192 5820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:41.640345 5820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:41.641405 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "a1560c596bc64c6ba4cbaf7f61519653"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "236a5021c8b9ad6cfde461f62332072e"
server_key_iv: "6f871e18d30a9b345b675227fc8e2d52"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.641768 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "a1560c596bc64c6ba4cbaf7f61519653"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "236a5021c8b9ad6cfde461f62332072e"
server_key_iv: "6f871e18d30a9b345b675227fc8e2d52"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.643165 5820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:53:41.643909 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.644083 5820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:41.644182 5820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "a1560c596bc64c6ba4cbaf7f61519653"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "236a5021c8b9ad6cfde461f62332072e"
server_key_iv: "6f871e18d30a9b345b675227fc8e2d52"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.644261 5820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:41.655205 5820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:41.657196 5820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:41.657356 5820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:41.657590 5820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:41.657898 5820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:41.657954 5820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.657995 5820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:41.658021 5820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.664103 5820 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:39817
I20260929 18:53:41.664145 5948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:39817 every 8 connection(s)
I20260929 18:53:41.664470 5820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:53:41.671334 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 5820
I20260929 18:53:41.671448 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:53:41.671644 29982 external_mini_cluster.cc:1428] Setting key 09407a0be2938746d7ce4bdc09182d04
I20260929 18:53:41.674052 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:41.665177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:39749 (local address 127.29.71.190:42219)
0929 18:53:41.665220 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:41.665222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:41.665436 (+ 214us) server_negotiation.cc:407] Connection header received
0929 18:53:41.665935 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:41.665936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:41.665949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:41.666002 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:41.666987 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.667498 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.668464 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.668673 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.668733 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:41.670794 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:41.670796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:41.670796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:41.670805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:41.672492 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.672977 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.672977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.672978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.673033 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.673328 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.673328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.673329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.673433 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:41.673483 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:41.673879 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 18:53:41.673984 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:53:41.674566 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:41.665180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:39749)
0929 18:53:41.665326 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:41.665363 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:41.665839 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:41.666061 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:41.666064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:41.666708 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:41.666906 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:41.666910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.667582 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.667583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:41.668378 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:41.668379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.668753 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.668754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:41.668854 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:41.669309 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:41.669317 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:41.670707 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:41.672564 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:41.672573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:41.672575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:41.672897 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:41.673114 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:41.673115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:41.673116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:41.673263 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:41.673493 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:41.673499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:41.673689 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 18:53:41.674237 (+ 548us) client_negotiation.cc:240] Negotiation successful
0929 18:53:41.674391 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260929 18:53:41.675094 5949 heartbeater.cc:347] Connected to a master server at 127.29.71.190:42219
I20260929 18:53:41.675205 5949 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:41.675413 5949 heartbeater.cc:511] Master 127.29.71.190:42219 requested a full tablet report, sending...
I20260929 18:53:41.675961 5758 ts_manager.cc:194] Registered new tserver with Master: a1560c596bc64c6ba4cbaf7f61519653 (127.29.71.129:39817)
I20260929 18:53:41.676945 5758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:39749
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:53:41.700810 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42219
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:41.783070 5954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:41.783286 5954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:41.783321 5954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:41.784504 5954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:41.784569 5954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:41.784614 5954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:41.786044 5954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:41.786326 5954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:41.786543 5954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:41.791424 5962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:41.791410 5960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:41.791481 5954 server_base.cc:1034] running on GCE node
W20260929 18:53:41.791406 5959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:41.791858 5954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:41.792106 5954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:41.793256 5954 hybrid_clock.cc:648] HybridClock initialized: now 1790708021793235 us; error 36 us; skew 500 ppm
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:41.795809 5954 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:53:41.796244 5954 webserver.cc:460] Webserver started at http://127.29.71.130:35701/ using document root <none> and password file <none>
I20260929 18:53:41.796437 5954 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:41.796478 5954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:41.796576 5954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:41.797560 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "515cf3b218264e5a8bde1251b432c25b"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "ccd279ff0205f483ae72bcead5632ffa"
server_key_iv: "a1f256f997898e062c444d5ceb65ee31"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.797860 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "515cf3b218264e5a8bde1251b432c25b"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "ccd279ff0205f483ae72bcead5632ffa"
server_key_iv: "a1f256f997898e062c444d5ceb65ee31"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.799170 5954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:53:41.799825 5969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.799988 5954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:41.800074 5954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "515cf3b218264e5a8bde1251b432c25b"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "ccd279ff0205f483ae72bcead5632ffa"
server_key_iv: "a1f256f997898e062c444d5ceb65ee31"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.800144 5954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:41.815900 5954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:41.818338 5954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:41.818496 5954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:41.818725 5954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:41.819061 5954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:41.819116 5954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.819156 5954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:41.819183 5954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.824882 5954 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:45663
I20260929 18:53:41.824928 6082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:45663 every 8 connection(s)
I20260929 18:53:41.825246 5954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:41.825486 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 5954
I20260929 18:53:41.825603 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:53:41.825814 29982 external_mini_cluster.cc:1428] Setting key e6f853d5282fdea9845896c0ff4905d0
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
I20260929 18:53:41.836370 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:41.825981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:43785 (local address 127.29.71.190:42219)
0929 18:53:41.826038 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:41.826040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:41.826307 (+ 267us) server_negotiation.cc:407] Connection header received
0929 18:53:41.826999 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:41.827000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:41.827010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:41.827050 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:41.828366 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.828872 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.830461 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.830659 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.830718 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:41.833438 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:41.833439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:41.833440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:41.833446 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:41.834956 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.835389 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.835390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.835391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.835458 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.835710 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.835711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.835712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.835828 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:41.835908 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:41.836194 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:53:41.836278 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:53:41.836645 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:41.825976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:43785)
0929 18:53:41.826145 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:41.826218 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:41.826893 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:41.827123 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:41.827127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:41.827955 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:41.828262 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:41.828274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.828970 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.828971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:41.830362 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:41.830364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.830755 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.830756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:41.830902 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:41.831593 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:41.831604 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:41.833280 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:41.835055 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:41.835062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:41.835064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:41.835285 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:41.835525 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:41.835526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:41.835526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:41.835642 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:41.835910 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:41.835917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:41.836055 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:53:41.836157 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:53:41.836411 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260929 18:53:41.837162 6083 heartbeater.cc:347] Connected to a master server at 127.29.71.190:42219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
I20260929 18:53:41.837266 6083 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:41.837463 6083 heartbeater.cc:511] Master 127.29.71.190:42219 requested a full tablet report, sending...
I20260929 18:53:41.837904 5758 ts_manager.cc:194] Registered new tserver with Master: 515cf3b218264e5a8bde1251b432c25b (127.29.71.130:45663)
I20260929 18:53:41.838454 5758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:43785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:53:41.853327 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42219
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:41.934346 6088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:41.934562 6088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:41.934587 6088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:41.935699 6088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:41.935731 6088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:41.935757 6088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:41.937119 6088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:41.937306 6088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:41.937482 6088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:41.942608 6093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:41.942646 6096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:41.942612 6094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:41.942858 6088 server_base.cc:1034] running on GCE node
I20260929 18:53:41.943140 6088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:41.943389 6088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:41.944536 6088 hybrid_clock.cc:648] HybridClock initialized: now 1790708021944522 us; error 30 us; skew 500 ppm
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:41.947065 6088 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:53:41.947518 6088 webserver.cc:460] Webserver started at http://127.29.71.131:33411/ using document root <none> and password file <none>
I20260929 18:53:41.947751 6088 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:41.947831 6088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:41.947983 6088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:41.949118 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "e44162ac592a4db7a56a975012a99b24"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "a5164b27722abf01c6731bf4241c2d0f"
server_key_iv: "436851a914900d898b346515cd8f6f9f"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.949482 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "e44162ac592a4db7a56a975012a99b24"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "a5164b27722abf01c6731bf4241c2d0f"
server_key_iv: "436851a914900d898b346515cd8f6f9f"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.950896 6088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:53:41.951674 6103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.951885 6088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:41.952004 6088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "e44162ac592a4db7a56a975012a99b24"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "a5164b27722abf01c6731bf4241c2d0f"
server_key_iv: "436851a914900d898b346515cd8f6f9f"
server_key_version: "encryptionkey@0"
I20260929 18:53:41.952085 6088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:41.961725 6088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:41.964138 6088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:41.964309 6088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:41.964563 6088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:41.964859 6088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:41.964910 6088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.964946 6088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:41.964993 6088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:41.970670 6088 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:42149
I20260929 18:53:41.970747 6216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:42149 every 8 connection(s)
I20260929 18:53:41.971182 6088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:53:41 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708021, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:53:41.978389 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6088
I20260929 18:53:41.978521 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:53:41.978742 29982 external_mini_cluster.cc:1428] Setting key 8f3c610d5800952bec5931de0e360725
I20260929 18:53:41.981051 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:41.972078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:58315 (local address 127.29.71.190:42219)
0929 18:53:41.972130 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:41.972133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:41.972265 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:53:41.972864 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:41.972865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:41.972877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:41.972935 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:41.973929 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.974514 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.975538 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.975738 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.975832 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:41.978013 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:41.978014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:41.978015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:41.978023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:41.979616 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.980057 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.980059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.980059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.980118 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:41.980397 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:41.980399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:41.980399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:41.980521 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:41.980597 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:41.980878 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 18:53:41.980956 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:53:41.981254 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:41.971912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:58315)
0929 18:53:41.972105 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:41.972158 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:41.972741 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:41.973004 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:41.973009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:41.973590 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:41.973785 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:41.973790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.974615 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.974616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:41.975439 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:41.975440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:41.975833 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:41.975834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:41.975970 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:41.976533 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:41.976541 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:41.977913 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:41.979726 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:41.979733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:41.979735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:41.979965 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:41.980195 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:53:41.980196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:53:41.980196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:53:41.980326 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:53:41.980586 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:41.980592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:41.980739 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 18:53:41.980857 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:53:41.981019 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260929 18:53:41.981793 6217 heartbeater.cc:347] Connected to a master server at 127.29.71.190:42219
I20260929 18:53:41.981909 6217 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:41.982113 6217 heartbeater.cc:511] Master 127.29.71.190:42219 requested a full tablet report, sending...
I20260929 18:53:41.982576 5758 ts_manager.cc:194] Registered new tserver with Master: e44162ac592a4db7a56a975012a99b24 (127.29.71.131:42149)
I20260929 18:53:41.983079 5758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:58315
I20260929 18:53:41.990756 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab.
Entry for principal oryx/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab.
WARNING: no policy specified for oryx/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab.
Entry for principal oryx/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab.
WARNING: no policy specified for oryx/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab.
Entry for principal oryx/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab.
I20260929 18:53:42.068748 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 5820
I20260929 18:53:42.073719 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 5954
I20260929 18:53:42.077899 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6088
I20260929 18:53:42.082593 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 5726
I20260929 18:53:42.088245 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42219
--webserver_interface=127.29.71.190
--webserver_port=45115
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:53:42.170496 6226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:42.170717 6226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:42.170740 6226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:42.172102 6226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:42.172156 6226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:42.172173 6226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:42.172185 6226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:42.172197 6226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:42.173708 6226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=45115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:42.173974 6226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:42.174242 6226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:42.179136 6232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:42.179147 6231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:42.179206 6234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:42.179299 6226 server_base.cc:1034] running on GCE node
I20260929 18:53:42.179520 6226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:42.179800 6226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:42.180938 6226 hybrid_clock.cc:648] HybridClock initialized: now 1790708022180918 us; error 36 us; skew 500 ppm
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708022, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:42.183372 6226 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:53:42.183844 6226 webserver.cc:460] Webserver started at http://127.29.71.190:45115/ using document root <none> and password file <none>
I20260929 18:53:42.184083 6226 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:42.184166 6226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:42.185529 6226 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:42.186256 6241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.186431 6226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:42.186527 6226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "887b5068543c4d4ab3b85daf020029de"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "2fc9ea3bbd8271a9aabe64efc80b302f"
server_key_iv: "490b841aac0d1d75ff7b5c8c43b8a883"
server_key_version: "encryptionkey@0"
I20260929 18:53:42.186878 6226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:42.201822 6226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:42.203783 6226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:42.203948 6226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:42.207949 6226 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:42219
I20260929 18:53:42.207985 6293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:42219 every 8 connection(s)
I20260929 18:53:42.208402 6226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:42.209828 6294 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:53:42.210515 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Bootstrap starting.
I20260929 18:53:42.213095 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6226
I20260929 18:53:42.213383 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:39817
--local_ip_for_outbound_sockets=127.29.71.129
--tserver_master_addrs=127.29.71.190:42219
--webserver_port=36565
--webserver_interface=127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:53:42.218778 6294 log.cc:824] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:42.220484 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:53:42.220786 6294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Bootstrap complete.
I20260929 18:53:42.223132 6294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:42.223351 6294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887b5068543c4d4ab3b85daf020029de, State: Initialized, Role: FOLLOWER
I20260929 18:53:42.223526 6294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:42.223593 6294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:42.223644 6294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:42.223693 6294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:53:42.224660 6294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:42.224784 6294 leader_election.cc:302] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 887b5068543c4d4ab3b85daf020029de; no voters:
I20260929 18:53:42.224946 6294 leader_election.cc:288] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:53:42.224978 6298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:53:42.225137 6298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [term 2 LEADER]: Becoming Leader. State: Replica: 887b5068543c4d4ab3b85daf020029de, State: Running, Role: LEADER
I20260929 18:53:42.225209 6294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:42.225303 6298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } }
I20260929 18:53:42.225749 6300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 887b5068543c4d4ab3b85daf020029de. Latest consensus state: current_term: 2 leader_uuid: "887b5068543c4d4ab3b85daf020029de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } } }
I20260929 18:53:42.225761 6299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "887b5068543c4d4ab3b85daf020029de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887b5068543c4d4ab3b85daf020029de" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42219 } } }
I20260929 18:53:42.225883 6299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:42.225839 6300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:42.226140 6306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:42.226718 6306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:42.226905 6306 catalog_manager.cc:1259] Loaded cluster ID: 685c8b7a4b8144509b404bbc0db25d8d
I20260929 18:53:42.227077 6306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:53:42.228368 6306 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:42.228605 6306 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 887b5068543c4d4ab3b85daf020029de: Loaded TSK: 0
I20260929 18:53:42.228899 6306 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:53:42.295481 6296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:42.295713 6296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:42.295755 6296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:42.296942 6296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:42.297008 6296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:42.297063 6296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:42.298616 6296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.129.keytab
--principal=oryx/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:39817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=36565
--tserver_master_addrs=127.29.71.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:42.298883 6296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:42.299121 6296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:42.305055 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:42.305073 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:42.305248 6296 server_base.cc:1034] running on GCE node
W20260929 18:53:42.305073 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:42.305685 6296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:42.305935 6296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:42.307093 6296 hybrid_clock.cc:648] HybridClock initialized: now 1790708022307070 us; error 42 us; skew 500 ppm
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708022, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:42.309684 6296 init.cc:377] Logged in from keytab as oryx/127.29.71.129@KRBTEST.COM (short username oryx)
I20260929 18:53:42.310128 6296 webserver.cc:460] Webserver started at http://127.29.71.129:36565/ using document root <none> and password file <none>
I20260929 18:53:42.310396 6296 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:42.310480 6296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:42.311889 6296 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:42.312521 6330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.312706 6296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:42.312817 6296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "a1560c596bc64c6ba4cbaf7f61519653"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "236a5021c8b9ad6cfde461f62332072e"
server_key_iv: "6f871e18d30a9b345b675227fc8e2d52"
server_key_version: "encryptionkey@0"
I20260929 18:53:42.313176 6296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:42.336416 6296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:42.338639 6296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:42.338811 6296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:42.339073 6296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:42.339383 6296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:42.339438 6296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.339478 6296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:42.339506 6296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.346832 6296 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:39817
I20260929 18:53:42.346873 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:39817 every 8 connection(s)
I20260929 18:53:42.347214 6296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:53:42.348332 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6296
I20260929 18:53:42.348625 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:45663
--local_ip_for_outbound_sockets=127.29.71.130
--tserver_master_addrs=127.29.71.190:42219
--webserver_port=35701
--webserver_interface=127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:53:42.356777 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.348002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:55739 (local address 127.29.71.190:42219)
0929 18:53:42.348199 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.348210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.348252 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:53:42.348953 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.348959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.349027 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.349151 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.350511 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.351590 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.352565 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.353273 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.353403 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.356650 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:55739: BlockingRecv error: recv got EOF from 127.29.71.129:55739 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":68,"threads_started":1}
I20260929 18:53:42.356791 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.347879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:55739)
0929 18:53:42.348089 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.348138 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.348705 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.349231 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.349235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.350062 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.350382 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.350389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.351690 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.351691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.352479 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.352482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.353367 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.353368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.353468 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.354029 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.354037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.356527 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":109,"threads_started":1}
W20260929 18:53:42.357005 6444 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:42219 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.71.190:42219: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:42.361930 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.357262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:34063)
0929 18:53:42.357310 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.357320 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.357359 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.357474 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.357475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.357636 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.357692 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.357695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.358366 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.358367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.359280 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.359283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.359612 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.359613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.359688 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.360071 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.360076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.361861 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 18:53:42.362030 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.357324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:34063 (local address 127.29.71.190:42219)
0929 18:53:42.357372 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.357374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.357386 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:53:42.357410 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.357411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.357420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.357468 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.357766 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.358293 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.359360 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.359538 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.359587 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.361940 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:34063: BlockingRecv error: recv got EOF from 127.29.71.129:34063 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:42.366947 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.362222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:38375)
0929 18:53:42.362296 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.362306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.362337 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.362532 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.362533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.362665 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.362698 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.362699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.363368 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.363369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.364174 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.364176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.364545 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.364546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.364599 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.364968 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.364973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.366879 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 18:53:42.367079 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.362255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:38375 (local address 127.29.71.190:42219)
0929 18:53:42.362316 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.362318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.362341 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:53:42.362396 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.362397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.362406 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.362445 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.362798 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.363292 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.364271 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.364459 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.364536 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.366992 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:38375: BlockingRecv error: recv got EOF from 127.29.71.129:38375 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 18:53:42.367149 6444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:53:42.433844 6448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:42.434065 6448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:42.434100 6448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:42.435305 6448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:42.435370 6448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:42.435415 6448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:42.436844 6448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.130.keytab
--principal=oryx/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:45663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=35701
--tserver_master_addrs=127.29.71.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:42.437091 6448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:42.437315 6448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:42.442258 6454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:42.442333 6448 server_base.cc:1034] running on GCE node
W20260929 18:53:42.442271 6456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:42.442306 6453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:42.442678 6448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:42.442888 6448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:42.444031 6448 hybrid_clock.cc:648] HybridClock initialized: now 1790708022444021 us; error 33 us; skew 500 ppm
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708022, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:42.446463 6448 init.cc:377] Logged in from keytab as oryx/127.29.71.130@KRBTEST.COM (short username oryx)
I20260929 18:53:42.446879 6448 webserver.cc:460] Webserver started at http://127.29.71.130:35701/ using document root <none> and password file <none>
I20260929 18:53:42.447098 6448 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:42.447150 6448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:42.448536 6448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:42.449134 6463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.449293 6448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.449394 6448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "515cf3b218264e5a8bde1251b432c25b"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "ccd279ff0205f483ae72bcead5632ffa"
server_key_iv: "a1f256f997898e062c444d5ceb65ee31"
server_key_version: "encryptionkey@0"
I20260929 18:53:42.449807 6448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:42.459553 6448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:42.461383 6448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:42.461550 6448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:42.461783 6448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:42.462100 6448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:42.462154 6448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.462218 6448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:42.462247 6448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.468176 6448 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:45663
I20260929 18:53:42.468225 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:45663 every 8 connection(s)
I20260929 18:53:42.468562 6448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:42.474213 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6448
I20260929 18:53:42.474529 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:42149
--local_ip_for_outbound_sockets=127.29.71.131
--tserver_master_addrs=127.29.71.190:42219
--webserver_port=33411
--webserver_interface=127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:53:42.476387 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.469261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:43305 (local address 127.29.71.190:42219)
0929 18:53:42.469332 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.469334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.469552 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:53:42.470124 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.470125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.470141 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.470221 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.471185 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.471689 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.472729 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.472929 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.473015 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.476340 (+ 3325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:43305: BlockingRecv error: recv got EOF from 127.29.71.130:43305 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:53:42.476497 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.469248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:43305)
0929 18:53:42.469430 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.469466 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.470012 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.470297 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.470300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.470875 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.471070 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.471074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.471796 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.471797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.472623 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.472626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.473025 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.473026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.473153 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.473715 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.473724 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.476247 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
W20260929 18:53:42.476751 6577 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:42219 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.71.190:42219: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:42.481472 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.476975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:41789)
0929 18:53:42.477026 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.477037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.477082 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.477223 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.477224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.477355 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.477397 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.477399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.478094 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.478095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.478653 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.478655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.479025 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.479026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.479089 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.479578 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.479582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.481388 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 18:53:42.481536 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.477029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:41789 (local address 127.29.71.190:42219)
0929 18:53:42.477070 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.477072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.477082 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:42.477139 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.477140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.477149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.477222 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.477477 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.478009 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.478747 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.478941 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.479031 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.481460 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:41789: BlockingRecv error: recv got EOF from 127.29.71.130:41789 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:42.486513 6579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.481838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:54667)
0929 18:53:42.481935 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.481948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.482008 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.482152 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.482152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.482339 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.482381 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.482383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.482978 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.482979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.483873 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.483876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.484202 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.484203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.484266 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.484663 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.484669 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.486440 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 18:53:42.486636 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.481886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:54667 (local address 127.29.71.190:42219)
0929 18:53:42.481944 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.481946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.482001 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:53:42.482072 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.482072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.482080 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.482124 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.482455 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.482898 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.483962 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.484132 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.484176 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.486514 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:54667: BlockingRecv error: recv got EOF from 127.29.71.130:54667 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 18:53:42.486776 6577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:53:42.558144 6580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:42.558367 6580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:42.558391 6580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:42.559587 6580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:42.559635 6580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:42.559664 6580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:42.561244 6580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/krb5kdc/oryx_127.29.71.131.keytab
--principal=oryx/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:42149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=33411
--tserver_master_addrs=127.29.71.190:42219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:42.561492 6580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:42.561722 6580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:42.567092 6585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:42.567189 6580 server_base.cc:1034] running on GCE node
W20260929 18:53:42.567094 6586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:42.567092 6588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:42.567574 6580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:42.567859 6580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:42.569041 6580 hybrid_clock.cc:648] HybridClock initialized: now 1790708022569017 us; error 34 us; skew 500 ppm
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708022, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:53:42.571796 6580 init.cc:377] Logged in from keytab as oryx/127.29.71.131@KRBTEST.COM (short username oryx)
I20260929 18:53:42.572335 6580 webserver.cc:460] Webserver started at http://127.29.71.131:33411/ using document root <none> and password file <none>
I20260929 18:53:42.572582 6580 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:42.572669 6580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:42.574085 6580 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:42.574766 6595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.574959 6580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:53:42.575080 6580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "e44162ac592a4db7a56a975012a99b24"
format_stamp: "Formatted at 2026-09-29 18:53:41 on dist-test-slave-q2kh"
server_key: "a5164b27722abf01c6731bf4241c2d0f"
server_key_iv: "436851a914900d898b346515cd8f6f9f"
server_key_version: "encryptionkey@0"
I20260929 18:53:42.575456 6580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:42.590277 6580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:42.592345 6580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:42.592528 6580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:42.592785 6580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:42.593118 6580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:42.593181 6580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.593225 6580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:42.593257 6580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:42.600118 6580 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:42149
I20260929 18:53:42.600155 6708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:42149 every 8 connection(s)
I20260929 18:53:42.600538 6580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:53:42.607865 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.601295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:54183 (local address 127.29.71.190:42219)
0929 18:53:42.601356 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.601358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.601631 (+ 273us) server_negotiation.cc:407] Connection header received
0929 18:53:42.602152 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.602153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.602191 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.602248 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.603201 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.603694 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.604683 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.604877 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.604953 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.607818 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:54183: BlockingRecv error: recv got EOF from 127.29.71.131:54183 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:53:42.607903 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.601288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:54183)
0929 18:53:42.601473 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.601536 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.602048 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.602317 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.602321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.602897 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.603091 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.603096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.603788 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.603789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.604581 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.604583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.604958 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.604959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.605092 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.605594 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.605602 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.607739 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
W20260929 18:53:42.608182 6709 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:42219 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.71.190:42219: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
I20260929 18:53:42.609836 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6580
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:42.613093 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.608423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:57759)
0929 18:53:42.608473 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.608484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.608531 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.608853 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.608854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.608985 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.609044 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.609045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.609721 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.609722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.610440 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.610442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.610810 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.610810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.610872 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.611349 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.611354 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.613024 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 18:53:42.613195 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.608577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:57759 (local address 127.29.71.190:42219)
0929 18:53:42.608641 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.608642 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.608648 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:53:42.608666 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.608667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.608675 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.608847 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.609130 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.609592 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.610528 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.610712 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.610798 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.613106 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:57759: BlockingRecv error: recv got EOF from 127.29.71.131:57759 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:42 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:42.617915 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.613322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:33555)
0929 18:53:42.613405 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:42.613433 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:42.613470 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:42.613687 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:42.613688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:42.613841 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:42.613873 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.613875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.614520 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.614521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.615016 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:42.615018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.615410 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.615411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:42.615462 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.615831 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:42.615835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:42.617869 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 18:53:42.618060 6709 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:53:42.618068 6447 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:42.613417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:33555 (local address 127.29.71.190:42219)
0929 18:53:42.613467 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.613469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.613482 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:53:42.613581 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.613582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.613591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.613655 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.613947 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.614434 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.615099 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.615327 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.615410 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.617998 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:33555: BlockingRecv error: recv got EOF from 127.29.71.131:33555 (error 108)
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":32}
I20260929 18:53:42.618140 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:42.610268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.29.71.190:42219)
0929 18:53:42.610421 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:42.610423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:42.610519 (+ 96us) server_negotiation.cc:407] Connection header received
0929 18:53:42.610610 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:42.610610 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:42.610620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:42.610674 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:42.611355 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.611840 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.612815 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:42.613113 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:42.613499 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:42.614576 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:42.614580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:42.614581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:53:42.614591 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:42.616536 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:42.616981 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:42.616990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:42.616990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:42.617085 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:53:42.617304 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:53:42.617305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:53:42.617305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:53:42.617552 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:42.617655 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:42.617818 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:53:42.617944 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
Sep 29 18:53:43 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:43 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:43.372061 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:43.367539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:45515)
0929 18:53:43.367671 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:43.367681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:43.367729 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:43.367919 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:43.367919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:43.368074 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:43.368121 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:43.368123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.368749 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.368750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:43.369295 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:43.369296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.369657 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.369658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:43.369699 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:43.370220 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:43.370226 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:43.371989 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
W20260929 18:53:43.372174 6715 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:43.367642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:45515 (local address 127.29.71.190:42219)
0929 18:53:43.367779 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:43.367781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:43.367795 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:53:43.367818 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:43.367818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:43.367827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:43.367901 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:43.368189 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.368680 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.369385 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.369578 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.369636 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:43.372092 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:45515: BlockingRecv error: recv got EOF from 127.29.71.129:45515 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
Sep 29 18:53:43 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:43 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:43.491988 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:43.487157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:39419)
0929 18:53:43.487314 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:43.487327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:43.487418 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:43.487593 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:43.487594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:43.487776 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:43.487820 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:43.487822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.488506 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.488507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:43.489043 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:43.489044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.489454 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.489455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:43.489514 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:43.490033 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:43.490045 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:43.491926 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
W20260929 18:53:43.492102 6715 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:43.487245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:39419 (local address 127.29.71.190:42219)
0929 18:53:43.487317 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:43.487319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:43.487397 (+ 78us) server_negotiation.cc:407] Connection header received
0929 18:53:43.487502 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:43.487503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:43.487516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:43.487589 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:43.487903 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.488421 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.489141 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.489335 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.489452 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:43.492033 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:39419: BlockingRecv error: recv got EOF from 127.29.71.130:39419 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 18:53:43 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:43 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:43.623152 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:43.618413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:50601)
0929 18:53:43.618565 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:43.618579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:43.618646 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:43.618818 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:43.618819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:43.618995 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:43.619044 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:43.619045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.619701 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.619702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:43.620250 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:43.620251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.620611 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.620612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:43.620674 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:43.621169 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:43.621174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:43.623053 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
W20260929 18:53:43.623229 6715 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:43.618528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:50601 (local address 127.29.71.190:42219)
0929 18:53:43.618599 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:43.618602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:43.618628 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:53:43.618731 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:43.618732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:43.618752 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:43.618812 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:43.619128 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.619618 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.620336 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:43.620527 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:43.620613 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:43.623169 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:50601: BlockingRecv error: recv got EOF from 127.29.71.131:50601 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:53:44 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:44 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:44.377615 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:44.372576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:42735)
0929 18:53:44.372730 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:44.372741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:44.372795 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:44.372946 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:44.372946 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:44.373118 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:44.373169 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:44.373171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.373867 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.373867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:44.374473 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:44.374475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.374943 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.374944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:44.375012 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:44.375583 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:44.375588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:44.377518 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
W20260929 18:53:44.377669 6719 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:44.372670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:42735 (local address 127.29.71.190:42219)
0929 18:53:44.372785 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:44.372786 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:44.372796 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:44.372853 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:44.372854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:44.372864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:44.372925 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:44.373262 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.373784 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.374548 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.374782 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.374940 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:44.377597 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:42735: BlockingRecv error: recv got EOF from 127.29.71.129:42735 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
Sep 29 18:53:44 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:44 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:44.497435 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:44.492516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:54957)
0929 18:53:44.492702 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:44.492714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:44.492804 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:44.492959 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:44.492960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:44.493160 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:44.493209 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:44.493211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.494112 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.494114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:44.494678 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:44.494679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.495071 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.495072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:44.495129 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:44.495590 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:44.495594 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:44.497342 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":114,"threads_started":1}
W20260929 18:53:44.497486 6719 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:44.492610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:54957 (local address 127.29.71.190:42219)
0929 18:53:44.492692 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:44.492694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:44.492785 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:53:44.492868 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:44.492869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:44.492881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:44.492953 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:44.493296 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.494018 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.494769 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.494979 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.495071 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:44.497425 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:54957: BlockingRecv error: recv got EOF from 127.29.71.130:54957 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:53:44 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:44 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:44.629418 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:44.623837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:43827)
0929 18:53:44.624051 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:44.624065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:44.624170 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:44.624358 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:44.624359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:44.624563 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:44.624633 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:44.624636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.625447 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.625448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:44.626143 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:44.626145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.626542 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.626543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:44.626599 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:44.627220 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:44.627224 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:44.629312 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":117,"threads_started":1}
W20260929 18:53:44.629513 6719 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:44.624082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:43827 (local address 127.29.71.190:42219)
0929 18:53:44.624158 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:44.624160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:44.624170 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:44.624244 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:44.624245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:44.624270 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:44.624337 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:44.624728 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.625369 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.626253 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:44.626457 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:44.626541 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:44.629430 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:43827: BlockingRecv error: recv got EOF from 127.29.71.131:43827 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 18:53:45 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:45 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:45.382680 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:45.378090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:37983)
0929 18:53:45.378275 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:45.378288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:45.378389 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:45.378608 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:45.378609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:45.378795 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:45.378843 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:45.378845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.379489 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.379489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:45.379996 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:45.379998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.380314 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.380315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:45.380355 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:45.380841 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:45.380846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:45.382609 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
W20260929 18:53:45.382810 6723 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:45.378213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:37983 (local address 127.29.71.190:42219)
0929 18:53:45.378377 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:45.378379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:45.378388 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:45.378472 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:45.378473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:45.378486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:45.378546 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:45.378912 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.379407 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.380064 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.380251 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.380300 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:45.382702 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:37983: BlockingRecv error: recv got EOF from 127.29.71.129:37983 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
Sep 29 18:53:45 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:45 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:45.502954 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:45.498015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:42271)
0929 18:53:45.498226 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:45.498240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:45.498323 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:45.498481 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:45.498482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:45.498732 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:45.498814 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:45.498817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.499543 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.499544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:45.500059 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:45.500061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.500497 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.500498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:45.500549 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:45.501036 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:45.501040 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:45.502884 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":128,"threads_started":1}
W20260929 18:53:45.503104 6723 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:45.498176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:42271 (local address 127.29.71.190:42219)
0929 18:53:45.498229 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:45.498231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:45.498346 (+ 115us) server_negotiation.cc:407] Connection header received
0929 18:53:45.498381 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:45.498382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:45.498395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:45.498459 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:45.498904 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.499461 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.500184 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.500414 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.500495 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:45.503002 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:42271: BlockingRecv error: recv got EOF from 127.29.71.130:42271 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 18:53:45 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:45 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:45.634879 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:45.630177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:52493)
0929 18:53:45.630297 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:45.630310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:45.630408 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:45.630537 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:45.630537 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:45.630711 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:45.630754 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:45.630756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.631427 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.631428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:45.631964 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:45.631965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.632390 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.632391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:45.632457 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:45.632982 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:45.632987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:45.634766 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260929 18:53:45.634913 6723 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:45.630206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:52493 (local address 127.29.71.190:42219)
0929 18:53:45.630256 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:45.630258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:45.630395 (+ 137us) server_negotiation.cc:407] Connection header received
0929 18:53:45.630465 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:45.630466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:45.630480 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:45.630532 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:45.630832 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.631341 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.632051 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:45.632298 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:45.632389 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:45.634848 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:52493: BlockingRecv error: recv got EOF from 127.29.71.131:52493 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 18:53:46 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:46 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:46.387850 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:46.383140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:42857)
0929 18:53:46.383301 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:46.383314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:46.383382 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:46.383529 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:46.383530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:46.383682 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:46.383731 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:46.383733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.384338 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.384339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:46.384889 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:46.384890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.385247 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.385248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:46.385300 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:46.385886 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:46.385890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:46.387793 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
W20260929 18:53:46.388016 6727 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:46.383193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:42857 (local address 127.29.71.190:42219)
0929 18:53:46.383346 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:46.383348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:46.383359 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:53:46.383441 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:46.383442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:46.383452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:46.383510 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:46.383790 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.384273 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.384962 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.385160 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.385217 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:46.387906 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:42857: BlockingRecv error: recv got EOF from 127.29.71.129:42857 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
Sep 29 18:53:46 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:46 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:46.511790 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:46.503544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:45163)
0929 18:53:46.503720 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:46.503731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:46.503816 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:46.503960 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:46.503961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:46.504155 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:46.504201 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:46.504203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.505041 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.505042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:46.505704 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:46.505707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.506026 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.506027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:46.506068 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:46.509778 (+ 3710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:46.509783 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:46.511729 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
W20260929 18:53:46.511947 6727 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:46.503602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:45163 (local address 127.29.71.190:42219)
0929 18:53:46.503673 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:46.503676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:46.503798 (+ 122us) server_negotiation.cc:407] Connection header received
0929 18:53:46.503882 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:46.503883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:46.503896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:46.503954 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:46.504283 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.504931 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.505772 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.505960 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.506015 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:46.511861 (+ 5846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:45163: BlockingRecv error: recv got EOF from 127.29.71.130:45163 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 18:53:46 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:46 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:46.640375 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:46.635499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:34275)
0929 18:53:46.635665 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:46.635678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:46.635769 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:46.635952 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:46.635953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:46.636132 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:46.636180 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:46.636182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.636904 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.636906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:46.637454 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:46.637455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.637823 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.637824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:46.637888 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:46.638460 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:46.638467 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:46.640275 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
W20260929 18:53:46.640436 6727 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:46.635586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:34275 (local address 127.29.71.190:42219)
0929 18:53:46.635684 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:46.635686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:46.635752 (+ 66us) server_negotiation.cc:407] Connection header received
0929 18:53:46.635834 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:46.635835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:46.635848 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:46.635937 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:46.636267 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.636801 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.637546 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:46.637734 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:46.637824 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:46.640361 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:34275: BlockingRecv error: recv got EOF from 127.29.71.131:34275 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 18:53:47 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:47 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:47.393487 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:47.388375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:36585)
0929 18:53:47.388515 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:47.388526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:47.388577 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:47.388780 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:47.388781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:47.388957 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:47.389007 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:47.389009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.389819 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.389821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:47.390583 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:47.390585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.390921 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.390922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:47.390962 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:47.391481 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:47.391486 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:47.393421 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260929 18:53:47.393608 6731 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:47.388468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:36585 (local address 127.29.71.190:42219)
0929 18:53:47.388617 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:47.388618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:47.388627 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:47.388651 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:47.388651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:47.388661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:47.388731 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:47.389099 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.389742 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.390656 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.390846 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.390904 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:47.393500 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:36585: BlockingRecv error: recv got EOF from 127.29.71.129:36585 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
Sep 29 18:53:47 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:47 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:53:47.517036 6731 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:47.512474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:53569 (local address 127.29.71.190:42219)
0929 18:53:47.512544 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:47.512547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:47.512555 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:47.512590 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:47.512591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:47.512601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:47.512659 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:47.513054 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.513557 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.514338 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.514537 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.514595 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:47.516984 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:53569: BlockingRecv error: recv got EOF from 127.29.71.130:53569 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:53:47.517050 6732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:47.512267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:53569)
0929 18:53:47.512425 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:47.512447 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:47.512508 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:47.512689 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:47.512689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:47.512907 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:47.512954 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:47.512955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.513633 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.513634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:47.514259 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:47.514262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.514602 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.514603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:47.514640 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:47.515122 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:47.515126 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:47.516905 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":101,"threads_started":1}
Sep 29 18:53:47 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:47 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:47.645776 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:47.640963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:45683)
0929 18:53:47.641125 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:47.641139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:47.641223 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:47.641390 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:47.641391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:47.641599 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:47.641648 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:47.641650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.642461 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.642462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:47.642999 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:47.643001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.643330 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.643331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:47.643379 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:47.643885 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:47.643889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:47.645719 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
W20260929 18:53:47.645926 6731 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:47.641078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:45683 (local address 127.29.71.190:42219)
0929 18:53:47.641143 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:47.641145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:47.641220 (+ 75us) server_negotiation.cc:407] Connection header received
0929 18:53:47.641289 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:47.641290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:47.641307 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:47.641360 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:47.641867 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.642384 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.643078 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:47.643253 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:47.643306 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:47.645809 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:45683: BlockingRecv error: recv got EOF from 127.29.71.131:45683 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 18:53:48 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:48 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:48.399293 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:48.394011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:55871)
0929 18:53:48.394223 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:48.394234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:48.394296 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:48.394433 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:48.394434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:48.394596 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:48.394646 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:48.394648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.395338 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.395339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:48.395989 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:48.395993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.396519 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.396521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:48.396600 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:48.397133 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:48.397138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:48.399180 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
W20260929 18:53:48.399362 6735 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:48.394102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:55871 (local address 127.29.71.190:42219)
0929 18:53:48.394240 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:48.394242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:48.394294 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:53:48.394351 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:48.394352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:48.394362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:48.394425 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:48.394709 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.395236 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.396164 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.396414 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.396513 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:48.399267 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:55871: BlockingRecv error: recv got EOF from 127.29.71.129:55871 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
Sep 29 18:53:48 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:48 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:48.522329 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:48.517539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:36297)
0929 18:53:48.517690 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:48.517703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:48.517773 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:48.518078 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:48.518080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:48.518313 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:48.518375 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:48.518377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.519043 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.519044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:48.519569 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:48.519571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.519926 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.519927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:48.519979 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:48.520490 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:48.520496 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:48.522273 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
W20260929 18:53:48.522491 6735 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:48.517586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:36297 (local address 127.29.71.190:42219)
0929 18:53:48.517633 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:48.517635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:48.517844 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:53:48.517872 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:48.517873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:48.517886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:48.517953 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:48.518449 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.518943 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.519642 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.519820 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.519870 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:48.522366 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:36297: BlockingRecv error: recv got EOF from 127.29.71.130:36297 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 18:53:48 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:48 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:53:48.651850 6735 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:48.646390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:53029 (local address 127.29.71.190:42219)
0929 18:53:48.646571 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:48.646574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:48.646688 (+ 114us) server_negotiation.cc:407] Connection header received
0929 18:53:48.646731 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:48.646732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:48.646745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:48.646795 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:48.647078 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.647641 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.648804 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.649034 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.649113 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:48.651798 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:53029: BlockingRecv error: recv got EOF from 127.29.71.131:53029 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260929 18:53:48.651862 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:48.646338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:53029)
0929 18:53:48.646612 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:48.646622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:48.646686 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:48.646810 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:48.646811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:48.646962 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:48.647010 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:48.647011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.647740 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.647741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:48.648722 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:48.648725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:48.649112 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:48.649112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:48.649167 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:48.649748 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:48.649755 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:48.651719 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":219,"threads_started":1}
Sep 29 18:53:49 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:49 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:49.404430 6738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:49.399877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:32835)
0929 18:53:49.400054 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:49.400070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:49.400141 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:49.400308 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:49.400309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:49.400469 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:49.400517 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:49.400519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.401173 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.401173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:49.401702 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:49.401704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.402106 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.402107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:49.402193 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:49.402743 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:49.402748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:49.404373 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":60,"threads_started":1}
W20260929 18:53:49.404574 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:49.400058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:32835 (local address 127.29.71.190:42219)
0929 18:53:49.400177 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:49.400179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:49.400187 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:49.400211 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:49.400211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:49.400220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:49.400283 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:49.400602 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.401106 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.401775 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.401984 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.402088 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:49.404485 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:32835: BlockingRecv error: recv got EOF from 127.29.71.129:32835 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
Sep 29 18:53:49 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:49 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:49.527379 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:49.522838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:51613)
0929 18:53:49.522991 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:49.523002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:49.523057 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:49.523179 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:49.523180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:49.523341 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:49.523389 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:49.523391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.523998 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.523998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:49.524509 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:49.524511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.524822 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.524823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:49.524861 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:49.525343 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:49.525347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:49.527285 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
W20260929 18:53:49.527427 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:49.522907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:51613 (local address 127.29.71.190:42219)
0929 18:53:49.522978 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:49.522980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:49.523059 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:53:49.523114 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:49.523115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:49.523124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:49.523167 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:49.523451 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.523932 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.524580 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.524755 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.524805 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:49.527363 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:51613: BlockingRecv error: recv got EOF from 127.29.71.130:51613 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 18:53:49 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:49 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:49.656800 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:49.652275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:37125)
0929 18:53:49.652411 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:49.652421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:49.652485 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:49.652650 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:49.652651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:49.652804 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:49.652850 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:49.652851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.653658 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.653659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:49.654254 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:49.654257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.654567 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.654568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:49.654605 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:49.655019 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:49.655037 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:49.656742 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260929 18:53:49.656898 6739 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:49.652324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:37125 (local address 127.29.71.190:42219)
0929 18:53:49.652402 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:49.652404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:49.652490 (+ 86us) server_negotiation.cc:407] Connection header received
0929 18:53:49.652553 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:49.652554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:49.652569 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:49.652642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:49.652961 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.653567 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.654330 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:49.654507 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:49.654561 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:49.656830 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:37125: BlockingRecv error: recv got EOF from 127.29.71.131:37125 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:53:50 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:50 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:50.409289 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:50.404884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:58195)
0929 18:53:50.405036 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:50.405048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:50.405105 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:50.405259 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:50.405260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:50.405420 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:50.405469 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:50.405471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.406114 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.406115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:50.406682 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:50.406684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.407011 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.407012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:50.407052 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:50.407471 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:50.407475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:50.409230 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
W20260929 18:53:50.409435 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:50.404998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:58195 (local address 127.29.71.190:42219)
0929 18:53:50.405138 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:50.405140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:50.405146 (+ 6us) server_negotiation.cc:407] Connection header received
0929 18:53:50.405167 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:50.405168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:50.405178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:50.405236 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:50.405553 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.406045 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.406753 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.406944 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.406997 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:50.409345 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:58195: BlockingRecv error: recv got EOF from 127.29.71.129:58195 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
Sep 29 18:53:50 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:50 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:50.532434 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:50.527832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:51517)
0929 18:53:50.527945 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:50.527959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:50.528023 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:50.528163 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:50.528163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:50.528321 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:50.528368 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:50.528370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.529132 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.529133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:50.529651 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:50.529652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.530011 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.530011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:50.530052 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:50.530559 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:50.530563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:50.532376 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
W20260929 18:53:50.532548 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:50.527907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:51517 (local address 127.29.71.190:42219)
0929 18:53:50.527985 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:50.527987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:50.528011 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:53:50.528083 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:50.528084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:50.528094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:50.528152 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:50.528430 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.529064 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.529734 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.529926 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.530002 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:50.532471 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:51517: BlockingRecv error: recv got EOF from 127.29.71.130:51517 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 18:53:50 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:50 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:50.661849 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:50.657267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:60505)
0929 18:53:50.657402 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:50.657413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:50.657492 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:50.657615 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:50.657616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:50.657773 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:50.657827 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:50.657828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.658524 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.658525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:50.659083 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:50.659086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.659421 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.659422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:50.659472 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:50.659959 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:50.659964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:50.661787 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
W20260929 18:53:50.661933 6743 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:50.657349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:60505 (local address 127.29.71.190:42219)
0929 18:53:50.657399 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:50.657401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:50.657480 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:53:50.657536 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:50.657537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:50.657548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:50.657594 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:50.657893 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.658439 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.659160 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:50.659338 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:50.659392 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:50.661884 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:60505: BlockingRecv error: recv got EOF from 127.29.71.131:60505 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 18:53:51 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:51 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:51.413988 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:51.409769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:58739)
0929 18:53:51.409880 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:51.409894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:51.409945 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:51.410079 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:51.410079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:51.410249 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:51.410297 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:51.410299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.410919 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.410919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:51.411427 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:51.411428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.411750 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.411751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:51.411791 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:51.412201 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:51.412205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:51.413908 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
W20260929 18:53:51.414115 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:51.409806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:58739 (local address 127.29.71.190:42219)
0929 18:53:51.409932 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:51.409934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:51.409942 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:53:51.409997 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:51.409998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:51.410008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:51.410067 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:51.410359 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.410853 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.411498 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.411687 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.411736 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:51.414023 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:58739: BlockingRecv error: recv got EOF from 127.29.71.129:58739 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
Sep 29 18:53:51 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:51 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:51.542006 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:51.532917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:44201)
0929 18:53:51.533051 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:51.533071 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:51.533129 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:51.533258 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:51.533259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:51.533419 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:51.533466 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:51.533468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.534094 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.534095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:51.534636 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:51.534638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.534962 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.534962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:51.535001 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:51.540015 (+ 5014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:51.540020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:51.541910 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260929 18:53:51.542054 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:51.533003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:44201 (local address 127.29.71.190:42219)
0929 18:53:51.533076 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:51.533077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:51.533131 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:53:51.533175 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:51.533175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:51.533185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:51.533240 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:51.533537 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.534027 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.534710 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.534895 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.534950 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:51.541994 (+ 7044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:44201: BlockingRecv error: recv got EOF from 127.29.71.130:44201 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 18:53:51 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:51 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:51.667164 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:51.662508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:54067)
0929 18:53:51.662671 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:51.662683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:51.662749 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:51.662891 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:51.662891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:51.663065 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:51.663112 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:51.663114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.663760 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.663761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:51.664349 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:51.664350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.664688 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.664689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:51.664748 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:51.665264 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:51.665268 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:51.667106 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
W20260929 18:53:51.667289 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:51.662576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:54067 (local address 127.29.71.190:42219)
0929 18:53:51.662632 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:51.662635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:51.662730 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:53:51.662799 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:51.662800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:51.662811 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:51.662857 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:51.663197 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.663683 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.664419 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:51.664598 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:51.664647 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:51.667193 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:54067: BlockingRecv error: recv got EOF from 127.29.71.131:54067 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 29 18:53:52 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:52 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:52.419312 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:52.414589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:56229)
0929 18:53:52.414741 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:52.414752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:52.414812 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:52.414943 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:52.414944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:52.415119 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:52.415168 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:52.415170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.415864 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.415866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:52.416498 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:52.416501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.416851 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.416852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:52.416892 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:52.417397 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:52.417402 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:52.419214 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
W20260929 18:53:52.419370 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:52.414659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:56229 (local address 127.29.71.190:42219)
0929 18:53:52.414757 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:52.414759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:52.414814 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:53:52.414855 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:52.414855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:52.414865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:52.414931 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:52.415231 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.415740 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.416588 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.416786 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.416840 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:52.419295 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:56229: BlockingRecv error: recv got EOF from 127.29.71.129:56229 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
Sep 29 18:53:52 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:52 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:52.547644 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:52.542539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:56477)
0929 18:53:52.542693 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:52.542715 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:52.542781 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:52.542935 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:52.542936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:52.543124 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:52.543172 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:52.543173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.544100 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.544101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:52.544636 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:52.544638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.544988 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.544989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:52.545028 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:52.545498 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:52.545502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:52.547540 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
W20260929 18:53:52.547694 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:52.542657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:56477 (local address 127.29.71.190:42219)
0929 18:53:52.542745 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:52.542747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:52.542761 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:53:52.542850 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:52.542850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:52.542862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:52.542916 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:52.543268 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.544009 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.544722 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.544918 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.544974 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:52.547627 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:56477: BlockingRecv error: recv got EOF from 127.29.71.130:56477 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 18:53:52 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:52 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:52.672286 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:52.667642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:37879)
0929 18:53:52.667825 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:52.667886 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:52.667954 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:52.668073 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:52.668073 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:52.668232 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:52.668277 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:52.668279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.668973 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.668974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:52.669509 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:52.669510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.669858 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.669858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:52.669931 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:52.670466 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:52.670470 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:52.672228 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
W20260929 18:53:52.672417 6751 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:52.667797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:37879 (local address 127.29.71.190:42219)
0929 18:53:52.667870 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:52.667872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:52.667949 (+ 77us) server_negotiation.cc:407] Connection header received
0929 18:53:52.668012 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:52.668013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:52.668025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:52.668061 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:52.668369 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.668879 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.669592 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:52.669789 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:52.669855 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:52.672335 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:37879: BlockingRecv error: recv got EOF from 127.29.71.131:37879 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 18:53:53 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:53 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:53.424767 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:53.419800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:48009)
0929 18:53:53.419968 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:53.419979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:53.420039 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:53.420178 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:53.420178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:53.420344 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:53.420391 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:53.420392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.421103 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.421105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:53.421955 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:53.421958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.422389 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.422390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:53.422460 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:53.422963 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:53.422968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:53.424707 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
W20260929 18:53:53.424949 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:53.419898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:48009 (local address 127.29.71.190:42219)
0929 18:53:53.420014 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:53.420016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:53.420025 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:53.420093 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:53.420093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:53.420102 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:53.420163 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:53.420458 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.421007 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.422033 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.422305 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.422378 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:53.424861 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:48009: BlockingRecv error: recv got EOF from 127.29.71.129:48009 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
Sep 29 18:53:53 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:53 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:53.553408 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:53.548343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:59317)
0929 18:53:53.548495 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:53.548506 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:53.548613 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:53.548777 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:53.548778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:53.549051 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:53.549107 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:53.549109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.549849 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.549851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:53.550428 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:53.550430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.550807 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.550808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:53.550864 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:53.551445 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:53.551451 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:53.553329 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
W20260929 18:53:53.553520 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:53.548442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:59317 (local address 127.29.71.190:42219)
0929 18:53:53.548499 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:53.548502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:53.548586 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:53:53.548679 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:53.548680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:53.548691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:53.548767 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:53.549197 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.549760 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.550515 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.550722 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.550807 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:53.553453 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:59317: BlockingRecv error: recv got EOF from 127.29.71.130:59317 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 29 18:53:53 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:53 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:53.678202 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:53.672917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:36907)
0929 18:53:53.673131 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:53.673144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:53.673246 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:53.673402 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:53.673403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:53.673598 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:53.673642 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:53.673645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.674462 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.674464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:53.674979 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:53.674981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.675418 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.675420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:53.675500 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:53.676047 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:53.676051 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:53.678072 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
W20260929 18:53:53.678380 6755 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:53.673021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:36907 (local address 127.29.71.190:42219)
0929 18:53:53.673123 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:53.673125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:53.673227 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:53:53.673328 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:53.673329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:53.673344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:53.673398 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:53.673729 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.674360 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.675067 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:53.675278 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:53.675362 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:53.678284 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:36907: BlockingRecv error: recv got EOF from 127.29.71.131:36907 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 18:53:54 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:54 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:54.430112 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:54.425214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:37231)
0929 18:53:54.425400 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:54.425413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:54.425480 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:54.425640 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:54.425641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:54.425860 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:54.425908 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:54.425910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.426775 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.426776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:54.427309 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:54.427311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.427692 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.427693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:54.427734 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:54.428187 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:54.428191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:54.430053 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
W20260929 18:53:54.430292 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:54.425343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:37231 (local address 127.29.71.190:42219)
0929 18:53:54.425469 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:54.425471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:54.425481 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:54.425549 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:54.425549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:54.425560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:54.425624 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:54.425981 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.426676 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.427409 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.427605 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.427667 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:54.430193 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:37231: BlockingRecv error: recv got EOF from 127.29.71.129:37231 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
Sep 29 18:53:54 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:54 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:54.559188 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:54.553915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:44313)
0929 18:53:54.554078 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:54.554088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:54.554135 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:54.554343 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:54.554344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:54.554506 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:54.554578 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:54.554580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.555534 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.555536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:54.556065 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:54.556067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.556430 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.556431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:54.556469 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:54.557036 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:54.557041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:54.559115 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
W20260929 18:53:54.559379 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:54.554011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:44313 (local address 127.29.71.190:42219)
0929 18:53:54.554109 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:54.554111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:54.554218 (+ 107us) server_negotiation.cc:407] Connection header received
0929 18:53:54.554251 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:54.554252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:54.554273 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:54.554331 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:54.554696 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.555459 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.556147 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.556347 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.556425 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:54.559273 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:44313: BlockingRecv error: recv got EOF from 127.29.71.130:44313 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 18:53:54 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:54 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:54.683501 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:54.678712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:57117)
0929 18:53:54.678885 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:54.678899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:54.678994 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:54.679183 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:54.679184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:54.679362 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:54.679405 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:54.679407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.680093 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.680094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:54.680615 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:54.680617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.680978 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.680979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:54.681047 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:54.681510 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:54.681514 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:54.683443 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
W20260929 18:53:54.683652 6759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:54.678810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:57117 (local address 127.29.71.190:42219)
0929 18:53:54.678884 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:54.678887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:54.678971 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:53:54.679085 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:54.679086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:54.679100 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:54.679174 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:54.679490 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.680006 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.680700 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:54.680894 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:54.680978 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:54.683547 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:57117: BlockingRecv error: recv got EOF from 127.29.71.131:57117 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 18:53:55 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:55 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:55.435751 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:55.430666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:43907)
0929 18:53:55.430829 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:55.430840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:55.430911 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:55.431133 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:55.431134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:55.431307 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:55.431356 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:55.431358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.432081 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.432082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:55.432653 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:55.432654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.433100 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.433100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:55.433141 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:55.433646 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:55.433650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:55.435650 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
W20260929 18:53:55.435813 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:55.430783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:43907 (local address 127.29.71.190:42219)
0929 18:53:55.430978 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:55.430980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:55.430989 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:55.431013 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:55.431013 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:55.431024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:55.431109 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:55.431442 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.431993 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.432739 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.433029 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.433087 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:55.435736 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:43907: BlockingRecv error: recv got EOF from 127.29.71.129:43907 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":134,"threads_started":1}
Sep 29 18:53:55 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:55 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:55.564483 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:55.559731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:52189)
0929 18:53:55.559873 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:55.559886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:55.559948 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:55.560117 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:55.560118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:55.560309 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:55.560365 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:55.560366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.561030 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.561031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:55.561611 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:55.561613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.562021 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.562022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:55.562076 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:55.562602 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:55.562607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:55.564386 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260929 18:53:55.564531 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:55.559876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:52189 (local address 127.29.71.190:42219)
0929 18:53:55.559988 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:55.559990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:55.559999 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:53:55.560032 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:55.560033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:55.560044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:55.560109 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:55.560446 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.560946 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.561717 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.561918 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.562023 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:55.564468 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:52189: BlockingRecv error: recv got EOF from 127.29.71.130:52189 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 18:53:55 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:55 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:55.688848 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:55.684072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:41565)
0929 18:53:55.684202 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:55.684214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:55.684278 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:55.684442 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:55.684443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:55.684701 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:55.684749 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:55.684751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.685419 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.685420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:55.686067 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:55.686069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.686453 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.686454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:55.686503 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:55.687033 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:55.687037 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:55.688772 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
W20260929 18:53:55.688977 6763 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:55.684122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:41565 (local address 127.29.71.190:42219)
0929 18:53:55.684209 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:55.684211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:55.684279 (+ 68us) server_negotiation.cc:407] Connection header received
0929 18:53:55.684327 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:55.684327 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:55.684341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:55.684417 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:55.684826 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.685335 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.686146 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:55.686369 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:55.686439 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:55.688901 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:41565: BlockingRecv error: recv got EOF from 127.29.71.131:41565 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:53:56 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:56 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:56.440908 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:56.436311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:55499)
0929 18:53:56.436450 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:56.436462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:56.436560 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:56.436829 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:56.436830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:56.437043 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:56.437099 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:56.437101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.437730 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.437731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:56.438277 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:56.438279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.438633 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.438633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:56.438672 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:56.439103 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:56.439107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:56.440853 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260929 18:53:56.441108 6767 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:56.436345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:55499 (local address 127.29.71.190:42219)
0929 18:53:56.436462 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:56.436464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:56.436690 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:53:56.436718 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:56.436718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:56.436734 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:56.436804 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:56.437167 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.437660 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.438366 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.438564 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.438619 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:56.440997 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:55499: BlockingRecv error: recv got EOF from 127.29.71.129:55499 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
Sep 29 18:53:56 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:56 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:56.574540 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:56.565019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:58749)
0929 18:53:56.565311 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:56.565323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:56.565413 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:56.565575 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:56.565576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:56.565758 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:56.565801 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:56.565802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.566526 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.566527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:56.567119 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:56.567121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.567491 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.567491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:56.567546 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:56.572543 (+ 4997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:56.572549 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:56.574445 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":201,"threads_started":1}
W20260929 18:53:56.574620 6767 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:56.565218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:58749 (local address 127.29.71.190:42219)
0929 18:53:56.565294 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:56.565296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:56.565395 (+ 99us) server_negotiation.cc:407] Connection header received
0929 18:53:56.565474 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:56.565475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:56.565487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:56.565565 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:56.565883 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.566446 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.567208 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.567407 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.567491 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:56.574559 (+ 7068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:58749: BlockingRecv error: recv got EOF from 127.29.71.130:58749 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 18:53:56 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:56 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.131@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:56.694065 6769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:56.689286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.131:44739)
0929 18:53:56.689463 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:56.689473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:56.689530 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:56.689665 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:56.689665 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:56.689824 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:56.689870 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:56.689872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.690691 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.690692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:56.691217 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:56.691219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.691554 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.691554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:56.691591 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:56.692105 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:56.692110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:56.694009 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
W20260929 18:53:56.694267 6767 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:56.689371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:44739 (local address 127.29.71.190:42219)
0929 18:53:56.689437 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:56.689439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:56.689517 (+ 78us) server_negotiation.cc:407] Connection header received
0929 18:53:56.689588 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:56.689589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:56.689602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:56.689644 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:56.690000 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.690610 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.691297 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:56.691485 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:56.691544 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:56.694158 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.131:44739: BlockingRecv error: recv got EOF from 127.29.71.131:44739 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 18:53:57 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:57 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.129@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:57.446300 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:57.441565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.129:48173)
0929 18:53:57.441717 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:57.441728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:57.441797 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:57.441936 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:57.441937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:57.442110 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:57.442219 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:57.442221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.443070 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.443071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:57.443626 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:57.443628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.443955 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.443956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:57.443993 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:57.444422 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:57.444427 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:57.446142 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260929 18:53:57.446434 6771 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:57.441613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:48173 (local address 127.29.71.190:42219)
0929 18:53:57.441728 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:57.441730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:57.441796 (+ 66us) server_negotiation.cc:407] Connection header received
0929 18:53:57.441842 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:57.441842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:57.441852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:57.441917 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:57.442299 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.442995 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.443695 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.443891 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.443942 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:57.446321 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.129:48173: BlockingRecv error: recv got EOF from 127.29.71.129:48173 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
Sep 29 18:53:57 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:53:57 dist-test-slave-q2kh krb5kdc[5713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.71.130@KRBTEST.COM for oryx/127.29.71.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:53:57.580170 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:57.575083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42219 (local address 127.29.71.130:49285)
0929 18:53:57.575273 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:57.575286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:57.575378 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:57.575546 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:57.575547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:57.575729 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:57.575773 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:57.575775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.576515 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.576516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:57.577061 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:57.577063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.577454 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.577455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:57.577511 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:57.578011 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:53:57.578024 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:57.580088 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.71.190:42219: Server oryx/127.29.71.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
W20260929 18:53:57.580279 6771 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:57.575204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:49285 (local address 127.29.71.190:42219)
0929 18:53:57.575275 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:57.575278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:57.575357 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:53:57.575450 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:57.575451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:57.575464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:57.575539 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:57.575898 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.576425 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.577159 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.577361 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.577454 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:57.580200 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.71.130:49285: BlockingRecv error: recv got EOF from 127.29.71.130:49285 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:53:57.614223 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6296
I20260929 18:53:57.618860 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6448
I20260929 18:53:57.623260 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6580
I20260929 18:53:57.627844 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6226
2026-09-29T18:53:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17437 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T18:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:57Z Disabled control of system clock
I20260929 18:53:57.656497 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35141
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:53:57.739984 6776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:57.740188 6776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:57.740208 6776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:57.741405 6776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:57.741446 6776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:57.741459 6776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:57.741472 6776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:57.741529 6776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:57.742865 6776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:35141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:57.743124 6776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:57.743435 6776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:57.748605 6782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:57.748598 6784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:57.748594 6781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:57.748800 6776 server_base.cc:1034] running on GCE node
I20260929 18:53:57.749042 6776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:57.749325 6776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:57.750481 6776 hybrid_clock.cc:648] HybridClock initialized: now 1790708037750469 us; error 25 us; skew 500 ppm
I20260929 18:53:57.751675 6776 webserver.cc:460] Webserver started at http://127.29.71.190:39687/ using document root <none> and password file <none>
I20260929 18:53:57.751953 6776 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:57.752033 6776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:57.752184 6776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:57.753225 6776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "a140e96d8e12454099e8806a7136cc1f"
format_stamp: "Formatted at 2026-09-29 18:53:57 on dist-test-slave-q2kh"
server_key: "93ab2ae3a92428c837e368aca6acb2cc"
server_key_iv: "d38ae3264ed016f04349294d4cb42522"
server_key_version: "encryptionkey@0"
I20260929 18:53:57.753595 6776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "a140e96d8e12454099e8806a7136cc1f"
format_stamp: "Formatted at 2026-09-29 18:53:57 on dist-test-slave-q2kh"
server_key: "93ab2ae3a92428c837e368aca6acb2cc"
server_key_iv: "d38ae3264ed016f04349294d4cb42522"
server_key_version: "encryptionkey@0"
I20260929 18:53:57.754954 6776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:53:57.755699 6790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:57.755872 6776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:53:57.755970 6776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "a140e96d8e12454099e8806a7136cc1f"
format_stamp: "Formatted at 2026-09-29 18:53:57 on dist-test-slave-q2kh"
server_key: "93ab2ae3a92428c837e368aca6acb2cc"
server_key_iv: "d38ae3264ed016f04349294d4cb42522"
server_key_version: "encryptionkey@0"
I20260929 18:53:57.756062 6776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:57.780223 6776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:57.780575 6776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:57.780740 6776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:57.784984 6776 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:35141
I20260929 18:53:57.785027 6842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:35141 every 8 connection(s)
I20260929 18:53:57.785403 6776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:57.786096 6843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:57.788976 6843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f: Bootstrap starting.
I20260929 18:53:57.789723 6843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:57.790045 6843 log.cc:824] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:57.790793 6843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f: No bootstrap required, opened a new log
I20260929 18:53:57.791260 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6776
I20260929 18:53:57.791399 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:57.791625 29982 external_mini_cluster.cc:1428] Setting key b98100c9830e02e21dc942868c8698e6
I20260929 18:53:57.793332 6843 raft_consensus.cc:348] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a140e96d8e12454099e8806a7136cc1f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35141 } }
I20260929 18:53:57.793535 6843 raft_consensus.cc:374] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:57.793577 6843 raft_consensus.cc:729] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a140e96d8e12454099e8806a7136cc1f, State: Initialized, Role: FOLLOWER
I20260929 18:53:57.793781 6843 consensus_queue.cc:260] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a140e96d8e12454099e8806a7136cc1f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35141 } }
I20260929 18:53:57.793896 6843 raft_consensus.cc:388] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:57.793942 6843 raft_consensus.cc:482] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:57.793999 6843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:57.794660 6843 raft_consensus.cc:504] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a140e96d8e12454099e8806a7136cc1f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35141 } }
I20260929 18:53:57.794787 6843 leader_election.cc:302] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a140e96d8e12454099e8806a7136cc1f; no voters:
I20260929 18:53:57.794955 6843 leader_election.cc:288] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:57.794986 6848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:57.795117 6848 raft_consensus.cc:686] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [term 1 LEADER]: Becoming Leader. State: Replica: a140e96d8e12454099e8806a7136cc1f, State: Running, Role: LEADER
I20260929 18:53:57.795240 6843 sys_catalog.cc:564] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:57.795274 6848 consensus_queue.cc:237] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a140e96d8e12454099e8806a7136cc1f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35141 } }
I20260929 18:53:57.795687 6848 sys_catalog.cc:455] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a140e96d8e12454099e8806a7136cc1f. Latest consensus state: current_term: 1 leader_uuid: "a140e96d8e12454099e8806a7136cc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a140e96d8e12454099e8806a7136cc1f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35141 } } }
I20260929 18:53:57.795697 6850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a140e96d8e12454099e8806a7136cc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a140e96d8e12454099e8806a7136cc1f" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35141 } } }
I20260929 18:53:57.795913 6850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:57.796133 6848 sys_catalog.cc:458] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:57.796543 6857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:57.797153 6857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:57.798980 6857 catalog_manager.cc:1347] Generated new cluster ID: 259a4629c8ff417aa2213d90cca50b60
I20260929 18:53:57.799048 6857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:53:57.810443 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:57.792226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.29.71.190:35141)
0929 18:53:57.792514 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:57.792524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:57.792561 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:53:57.793286 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:57.793295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:57.793385 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:57.793549 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:57.793836 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.795348 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.796485 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.797224 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.797415 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:57.798415 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:57.798419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:57.798419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:57.798628 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:57.798663 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:57.798675 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:57.798763 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:57.798836 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:53:57.798917 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":123,"threads_started":1}
I20260929 18:53:57.825119 6857 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:57.825934 6857 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:57.838680 6857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a140e96d8e12454099e8806a7136cc1f: Generated new TSK 0
I20260929 18:53:57.838948 6857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:53:57.840539 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35141
--builtin_ntp_servers=127.29.71.148:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:53:57.938951 6867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:57.939162 6867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:57.939183 6867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:57.940351 6867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:57.940393 6867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:57.940418 6867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:57.941735 6867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:57.941962 6867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:57.942293 6867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:57.947315 6872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:57.947379 6873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:57.947592 6867 server_base.cc:1034] running on GCE node
W20260929 18:53:57.947315 6875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:57.947888 6867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:57.948122 6867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:57.949280 6867 hybrid_clock.cc:648] HybridClock initialized: now 1790708037949267 us; error 32 us; skew 500 ppm
I20260929 18:53:57.950528 6867 webserver.cc:460] Webserver started at http://127.29.71.129:38683/ using document root <none> and password file <none>
I20260929 18:53:57.950768 6867 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:57.950841 6867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:57.950982 6867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:57.952039 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "e8ab71c452b345b4806e4d4a6e4047ee"
format_stamp: "Formatted at 2026-09-29 18:53:57 on dist-test-slave-q2kh"
server_key: "529f368273e4e69d05dcd2f3d9e55b07"
server_key_iv: "fac8f8edf041dd408bad047b52c767aa"
server_key_version: "encryptionkey@0"
I20260929 18:53:57.952404 6867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "e8ab71c452b345b4806e4d4a6e4047ee"
format_stamp: "Formatted at 2026-09-29 18:53:57 on dist-test-slave-q2kh"
server_key: "529f368273e4e69d05dcd2f3d9e55b07"
server_key_iv: "fac8f8edf041dd408bad047b52c767aa"
server_key_version: "encryptionkey@0"
I20260929 18:53:57.953763 6867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:53:57.954622 6881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:57.954808 6867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:53:57.954907 6867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "e8ab71c452b345b4806e4d4a6e4047ee"
format_stamp: "Formatted at 2026-09-29 18:53:57 on dist-test-slave-q2kh"
server_key: "529f368273e4e69d05dcd2f3d9e55b07"
server_key_iv: "fac8f8edf041dd408bad047b52c767aa"
server_key_version: "encryptionkey@0"
I20260929 18:53:57.954991 6867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:57.968067 6867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:57.968387 6867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:57.968544 6867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:57.968811 6867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:57.969175 6867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:57.969233 6867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:57.969266 6867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:57.969295 6867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:57.975018 6867 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:38903
I20260929 18:53:57.975059 6994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:38903 every 8 connection(s)
I20260929 18:53:57.975411 6867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:53:57.976145 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6867
I20260929 18:53:57.976244 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:53:57.976471 29982 external_mini_cluster.cc:1428] Setting key 78b51ca859ceccb72ff6f8d9f3cf712d
I20260929 18:53:57.977495 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35141
--builtin_ntp_servers=127.29.71.148:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:53:57.981485 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:57.976693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:40845 (local address 127.29.71.190:35141)
0929 18:53:57.976759 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:57.976762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:57.976772 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:53:57.977249 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:57.977250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:57.977262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:57.977322 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:57.978035 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.978829 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.979790 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.980138 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.980210 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:57.981105 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:57.981107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:57.981107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:57.981113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:57.981132 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:57.981137 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:57.981202 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:57.981331 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:53:57.981370 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:53:57.981577 6997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:57.976216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35141 (local address 127.29.71.129:40845)
0929 18:53:57.976418 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:57.976452 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:57.977131 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:57.977375 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:57.977379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:57.977386 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:57.977941 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:57.977946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.978913 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.978914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:57.979692 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:57.979694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:57.980223 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:57.980224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:57.980348 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:57.980923 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:53:57.980923 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:57.980969 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:53:57.980997 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:57.981224 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:57.981230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:57.981233 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:57.981306 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:53:57.981387 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260929 18:53:57.982090 6995 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35141
I20260929 18:53:57.982218 6995 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:57.982446 6995 heartbeater.cc:511] Master 127.29.71.190:35141 requested a full tablet report, sending...
I20260929 18:53:57.982959 6807 ts_manager.cc:194] Registered new tserver with Master: e8ab71c452b345b4806e4d4a6e4047ee (127.29.71.129:38903)
I20260929 18:53:57.983575 6807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.129:40845
W20260929 18:53:58.059873 6998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:58.060143 6998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:58.060181 6998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:58.061337 6998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:58.061408 6998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:58.061448 6998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:58.062899 6998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:58.063163 6998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:58.063467 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:58.068521 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.068625 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.068506 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:58.068760 6998 server_base.cc:1034] running on GCE node
I20260929 18:53:58.069061 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:58.069311 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:58.070462 6998 hybrid_clock.cc:648] HybridClock initialized: now 1790708038070440 us; error 46 us; skew 500 ppm
I20260929 18:53:58.071764 6998 webserver.cc:460] Webserver started at http://127.29.71.130:45713/ using document root <none> and password file <none>
I20260929 18:53:58.072062 6998 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:58.072113 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:58.072288 6998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:58.073385 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "7ad95e08edf843d48b2010253a18b93b"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "7bdf84230ff57081f033276a930c1212"
server_key_iv: "9ea4ad7685c10bd7448ecc4c0cb7c64d"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.073741 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "7ad95e08edf843d48b2010253a18b93b"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "7bdf84230ff57081f033276a930c1212"
server_key_iv: "9ea4ad7685c10bd7448ecc4c0cb7c64d"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.075101 6998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:53:58.075883 7012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.076107 6998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:53:58.076192 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "7ad95e08edf843d48b2010253a18b93b"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "7bdf84230ff57081f033276a930c1212"
server_key_iv: "9ea4ad7685c10bd7448ecc4c0cb7c64d"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.076264 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:58.102016 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:58.102406 6998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:58.102573 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:58.102821 6998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:58.103160 6998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:58.103194 6998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.103240 6998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:58.103266 6998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.108995 6998 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:33729
I20260929 18:53:58.109087 7125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:33729 every 8 connection(s)
I20260929 18:53:58.109427 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:58.112958 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 6998
I20260929 18:53:58.113061 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:53:58.113245 29982 external_mini_cluster.cc:1428] Setting key 51f5ae0925df5aabda190d40b9263838
I20260929 18:53:58.114224 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35141
--builtin_ntp_servers=127.29.71.148:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:53:58.115393 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.110208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:49109 (local address 127.29.71.190:35141)
0929 18:53:58.110251 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.110253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.110499 (+ 246us) server_negotiation.cc:407] Connection header received
0929 18:53:58.111011 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.111012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.111023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.111085 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.111776 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.112295 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.113605 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.113815 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.113902 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:58.115142 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:58.115144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:58.115144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:58.115148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:58.115164 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:58.115168 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:58.115218 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:58.115312 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:53:58.115335 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:53:58.115689 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.110151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35141 (local address 127.29.71.130:49109)
0929 18:53:58.110365 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:58.110398 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:58.110887 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:58.111162 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:58.111165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:58.111173 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:58.111670 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:58.111674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.112374 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.112375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:58.113491 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:58.113494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.114240 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.114242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:58.114418 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:58.114972 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:53:58.114973 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:58.115014 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:53:58.115051 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:58.115235 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:58.115240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:58.115243 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:58.115314 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:53:58.115429 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260929 18:53:58.116149 7126 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35141
I20260929 18:53:58.116237 7126 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:58.116441 7126 heartbeater.cc:511] Master 127.29.71.190:35141 requested a full tablet report, sending...
I20260929 18:53:58.116890 6807 ts_manager.cc:194] Registered new tserver with Master: 7ad95e08edf843d48b2010253a18b93b (127.29.71.130:33729)
I20260929 18:53:58.117437 6807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.130:49109
W20260929 18:53:58.196542 7129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:58.196769 7129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:58.196791 7129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:58.197950 7129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:58.197991 7129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:58.198024 7129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:58.199306 7129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:58.199494 7129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:58.199741 7129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:58.204660 7134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.204653 7135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.204658 7137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:58.204883 7129 server_base.cc:1034] running on GCE node
I20260929 18:53:58.205101 7129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:58.205308 7129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:58.206449 7129 hybrid_clock.cc:648] HybridClock initialized: now 1790708038206426 us; error 35 us; skew 500 ppm
I20260929 18:53:58.207717 7129 webserver.cc:460] Webserver started at http://127.29.71.131:37821/ using document root <none> and password file <none>
I20260929 18:53:58.207911 7129 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:58.207952 7129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:58.208055 7129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:58.209098 7129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "371d15b0249a4fdf91d3511732fe44a6"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "309ac18a2cf5b3cefed3b807c4cefd45"
server_key_iv: "63219d223bbb4c1a4ab6ad9b36a7f72a"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.209393 7129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "371d15b0249a4fdf91d3511732fe44a6"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "309ac18a2cf5b3cefed3b807c4cefd45"
server_key_iv: "63219d223bbb4c1a4ab6ad9b36a7f72a"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.210758 7129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:58.212888 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.213081 7129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:53:58.213171 7129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "371d15b0249a4fdf91d3511732fe44a6"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "309ac18a2cf5b3cefed3b807c4cefd45"
server_key_iv: "63219d223bbb4c1a4ab6ad9b36a7f72a"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.213229 7129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:58.241286 7129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:58.241586 7129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:58.241700 7129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:58.241912 7129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:58.242347 7129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:58.242388 7129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.242414 7129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:58.242466 7129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.248505 7129 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:46507
I20260929 18:53:58.248623 7256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:46507 every 8 connection(s)
I20260929 18:53:58.248956 7129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:53:58.249969 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 7129
I20260929 18:53:58.250079 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:53:58.250336 29982 external_mini_cluster.cc:1428] Setting key 1ab0eba006df99e4d4f9922deee4d76f
I20260929 18:53:58.254832 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.249729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:58379 (local address 127.29.71.190:35141)
0929 18:53:58.249777 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.249779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.250053 (+ 274us) server_negotiation.cc:407] Connection header received
0929 18:53:58.250844 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.250845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.250859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.250924 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.251618 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.252131 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.253230 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.253459 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.253553 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:58.254544 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:58.254546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:58.254546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:58.254551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:58.254567 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:58.254571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:58.254627 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:58.254721 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:53:58.254752 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:53:58.255100 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.249667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35141 (local address 127.29.71.131:58379)
0929 18:53:58.249917 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:58.249954 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:58.250710 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:58.251005 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:58.251009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:58.251017 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:58.251517 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:58.251521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.252251 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.252252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:58.253121 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:53:58.253123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:53:58.253563 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:53:58.253564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:53:58.253702 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:53:58.254381 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:53:58.254381 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:58.254411 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:53:58.254436 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:58.254629 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:58.254634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:58.254637 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:58.254725 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:53:58.254839 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260929 18:53:58.255558 7257 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35141
I20260929 18:53:58.255651 7257 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:58.255890 7257 heartbeater.cc:511] Master 127.29.71.190:35141 requested a full tablet report, sending...
I20260929 18:53:58.256271 6807 ts_manager.cc:194] Registered new tserver with Master: 371d15b0249a4fdf91d3511732fe44a6 (127.29.71.131:46507)
I20260929 18:53:58.256605 6807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.131:58379
I20260929 18:53:58.262121 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:53:58.264206 6847 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:58.263600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.29.71.190:35141)
0929 18:53:58.263704 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.263706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.263785 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:53:58.263888 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.263888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.263904 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.263977 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.264127 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42242: BlockingRecv error: recv got EOF from 127.0.0.1:42242 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:53:58.264295 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6867
I20260929 18:53:58.269024 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6998
I20260929 18:53:58.273154 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 7129
I20260929 18:53:58.277184 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 6776
2026-09-29T18:53:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (647 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T18:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:58Z Disabled control of system clock
I20260929 18:53:58.303296 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:46251
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:53:58.382579 7268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:58.382787 7268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:58.382808 7268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:58.383904 7268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:53:58.383946 7268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:58.383960 7268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:58.383972 7268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:53:58.383984 7268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:53:58.385243 7268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:46251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:58.385428 7268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:58.385649 7268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:58.390501 7273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.390509 7274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.390518 7276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:58.390904 7268 server_base.cc:1034] running on GCE node
I20260929 18:53:58.391198 7268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:58.391522 7268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:58.392668 7268 hybrid_clock.cc:648] HybridClock initialized: now 1790708038392654 us; error 28 us; skew 500 ppm
I20260929 18:53:58.393997 7268 webserver.cc:460] Webserver started at http://127.29.71.190:35521/ using document root <none> and password file <none>
I20260929 18:53:58.394307 7268 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:58.394378 7268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:58.394517 7268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:58.395578 7268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "7a6d34b310be44679f7cbd1dc3326b11"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "30167393beab95225eed107b33a3f355"
server_key_iv: "9c45ce22052f797022850bc8deb62d48"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.395955 7268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "7a6d34b310be44679f7cbd1dc3326b11"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "30167393beab95225eed107b33a3f355"
server_key_iv: "9c45ce22052f797022850bc8deb62d48"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.397308 7268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:58.398094 7282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.398317 7268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:53:58.398424 7268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "7a6d34b310be44679f7cbd1dc3326b11"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "30167393beab95225eed107b33a3f355"
server_key_iv: "9c45ce22052f797022850bc8deb62d48"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.398514 7268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:58.401862 7268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:58.402196 7268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:58.402338 7268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:58.406265 7268 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:46251
I20260929 18:53:58.406317 7334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:46251 every 8 connection(s)
I20260929 18:53:58.406651 7268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:53:58.407246 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 7268
I20260929 18:53:58.407362 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:53:58.407387 7335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:53:58.407574 29982 external_mini_cluster.cc:1428] Setting key 1a3c59b99481bf0874c73a511989d97f
I20260929 18:53:58.410503 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.408218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.29.71.190:46251)
0929 18:53:58.408424 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.408432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.408463 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:53:58.409143 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.409151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.409194 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.409314 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.410019 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:58.410022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:58.410022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:58.410026 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:58.410063 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:58.410073 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:58.410129 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:58.410243 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 18:53:58.410322 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":101,"threads_started":1}
I20260929 18:53:58.411484 7335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11: Bootstrap starting.
I20260929 18:53:58.412020 7335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11: Neither blocks nor log segments found. Creating new log.
I20260929 18:53:58.412257 7335 log.cc:824] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11: Log is configured to *not* fsync() on all Append() calls
I20260929 18:53:58.412943 7335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11: No bootstrap required, opened a new log
I20260929 18:53:58.414299 7335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6d34b310be44679f7cbd1dc3326b11" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46251 } }
I20260929 18:53:58.414444 7335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:53:58.414469 7335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6d34b310be44679f7cbd1dc3326b11, State: Initialized, Role: FOLLOWER
I20260929 18:53:58.414552 7335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6d34b310be44679f7cbd1dc3326b11" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46251 } }
I20260929 18:53:58.414601 7335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:53:58.414623 7335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:53:58.414654 7335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:53:58.415149 7335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6d34b310be44679f7cbd1dc3326b11" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46251 } }
I20260929 18:53:58.415230 7335 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a6d34b310be44679f7cbd1dc3326b11; no voters:
I20260929 18:53:58.415388 7335 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:53:58.415670 7340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:53:58.415681 7335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:53:58.415800 7340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [term 1 LEADER]: Becoming Leader. State: Replica: 7a6d34b310be44679f7cbd1dc3326b11, State: Running, Role: LEADER
I20260929 18:53:58.415892 7340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6d34b310be44679f7cbd1dc3326b11" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46251 } }
I20260929 18:53:58.416232 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a6d34b310be44679f7cbd1dc3326b11. Latest consensus state: current_term: 1 leader_uuid: "7a6d34b310be44679f7cbd1dc3326b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6d34b310be44679f7cbd1dc3326b11" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46251 } } }
I20260929 18:53:58.416319 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:58.416267 7341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a6d34b310be44679f7cbd1dc3326b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6d34b310be44679f7cbd1dc3326b11" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46251 } } }
I20260929 18:53:58.416728 7341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11 [sys.catalog]: This master's current role is: LEADER
I20260929 18:53:58.417061 7349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:53:58.417557 7349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:53:58.419083 7349 catalog_manager.cc:1347] Generated new cluster ID: 47266778062f418bbeaa2486c84935b3
I20260929 18:53:58.419137 7349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:53:58.434677 7349 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:53:58.434882 7349 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:53:58.442828 7349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a6d34b310be44679f7cbd1dc3326b11: Generated new TSK 0
I20260929 18:53:58.443046 7349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:53:58.444473 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46251
--builtin_ntp_servers=127.29.71.148:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:53:58.525951 7359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:58.526212 7359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:58.526250 7359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:58.527437 7359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:58.527475 7359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:58.527500 7359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:53:58.528915 7359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:58.529150 7359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:58.529455 7359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:58.535400 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.535439 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:58.535614 7359 server_base.cc:1034] running on GCE node
W20260929 18:53:58.535434 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:58.535890 7359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:58.536176 7359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:58.537333 7359 hybrid_clock.cc:648] HybridClock initialized: now 1790708038537312 us; error 40 us; skew 500 ppm
I20260929 18:53:58.538822 7359 webserver.cc:460] Webserver started at http://127.29.71.129:42599/ using document root <none> and password file <none>
I20260929 18:53:58.539125 7359 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:58.539188 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:58.539304 7359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:58.540616 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "6adae967a1234e91a2b1e0ff90509f62"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "bda8af1e855b9aee53e682ecd44f447a"
server_key_iv: "a069b6fb5aa8a5658d52c5f665e73671"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.541006 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "6adae967a1234e91a2b1e0ff90509f62"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "bda8af1e855b9aee53e682ecd44f447a"
server_key_iv: "a069b6fb5aa8a5658d52c5f665e73671"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.542623 7359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:58.543380 7373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.543555 7359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:58.543622 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "6adae967a1234e91a2b1e0ff90509f62"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "bda8af1e855b9aee53e682ecd44f447a"
server_key_iv: "a069b6fb5aa8a5658d52c5f665e73671"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.543671 7359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:58.546916 7359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:58.547123 7359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:58.547214 7359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:58.547363 7359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:58.547597 7359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:58.547626 7359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.547645 7359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:58.547657 7359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.553196 7359 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:45503
I20260929 18:53:58.553279 7486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:45503 every 8 connection(s)
I20260929 18:53:58.553596 7359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:53:58.556799 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.554414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:57113 (local address 127.29.71.190:46251)
0929 18:53:58.554467 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.554469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.554679 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:53:58.555241 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.555243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.555250 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.555309 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.556546 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:58.556548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:58.556548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:58.556567 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:58.556582 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:58.556586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:58.556637 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:58.556717 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 18:53:58.556732 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:53:58.557008 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.554380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46251 (local address 127.29.71.129:57113)
0929 18:53:58.554553 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:58.554586 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:58.555119 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:58.555377 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:58.555381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:58.555390 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:58.556391 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:53:58.556392 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:58.556421 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:53:58.556449 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:58.556635 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:58.556640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:58.556643 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:58.556717 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:53:58.556779 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260929 18:53:58.557482 7487 heartbeater.cc:347] Connected to a master server at 127.29.71.190:46251
I20260929 18:53:58.557579 7487 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:58.557735 7487 heartbeater.cc:511] Master 127.29.71.190:46251 requested a full tablet report, sending...
I20260929 18:53:58.558413 7299 ts_manager.cc:194] Registered new tserver with Master: 6adae967a1234e91a2b1e0ff90509f62 (127.29.71.129:45503)
I20260929 18:53:58.559347 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 7359
I20260929 18:53:58.559434 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:53:58.559669 29982 external_mini_cluster.cc:1428] Setting key 97828534af71b0c479cca8c6fe656e50
I20260929 18:53:58.560734 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46251
--builtin_ntp_servers=127.29.71.148:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:53:58.641716 7490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:58.641966 7490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:58.642002 7490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:58.643247 7490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:58.643319 7490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:58.643364 7490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:53:58.644731 7490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:58.644994 7490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:58.645287 7490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:58.650552 7496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.650590 7495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.650758 7498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:58.650749 7490 server_base.cc:1034] running on GCE node
I20260929 18:53:58.651013 7490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:58.651324 7490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:58.652508 7490 hybrid_clock.cc:648] HybridClock initialized: now 1790708038652483 us; error 33 us; skew 500 ppm
I20260929 18:53:58.653818 7490 webserver.cc:460] Webserver started at http://127.29.71.130:36217/ using document root <none> and password file <none>
I20260929 18:53:58.654094 7490 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:58.654186 7490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:58.654336 7490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:58.655539 7490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "a1f7afa395cc4d73bc7fa11360a0f26f"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "0221f25b1aafdb2f40a2bb0510500895"
server_key_iv: "7d55e35c5f24524ab0c35067e4dd6c23"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.655915 7490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "a1f7afa395cc4d73bc7fa11360a0f26f"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "0221f25b1aafdb2f40a2bb0510500895"
server_key_iv: "7d55e35c5f24524ab0c35067e4dd6c23"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.657197 7490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:53:58.657858 7504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.658025 7490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:58.658128 7490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "a1f7afa395cc4d73bc7fa11360a0f26f"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "0221f25b1aafdb2f40a2bb0510500895"
server_key_iv: "7d55e35c5f24524ab0c35067e4dd6c23"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.658246 7490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:58.661495 7490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:58.661752 7490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:58.661899 7490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:58.662102 7490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:58.662420 7490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:58.662473 7490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.662509 7490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:58.662535 7490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.668247 7490 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:34493
I20260929 18:53:58.668293 7617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:34493 every 8 connection(s)
I20260929 18:53:58.668586 7490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:53:58.671434 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.669237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:54599 (local address 127.29.71.190:46251)
0929 18:53:58.669277 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.669279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.669534 (+ 255us) server_negotiation.cc:407] Connection header received
0929 18:53:58.669968 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.669969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.669974 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.670034 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.671165 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:58.671166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:58.671167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:58.671171 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:58.671185 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:58.671189 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:58.671239 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:58.671370 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 18:53:58.671382 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:53:58.671661 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.669219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46251 (local address 127.29.71.130:54599)
0929 18:53:58.669406 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:58.669445 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:58.669877 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:58.670108 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:58.670111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:58.670119 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:58.671034 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:53:58.671035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:58.671060 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:53:58.671086 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:58.671274 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:58.671280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:58.671283 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:58.671374 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:53:58.671441 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260929 18:53:58.672161 7618 heartbeater.cc:347] Connected to a master server at 127.29.71.190:46251
I20260929 18:53:58.672235 7618 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:58.672379 7618 heartbeater.cc:511] Master 127.29.71.190:46251 requested a full tablet report, sending...
I20260929 18:53:58.672741 7299 ts_manager.cc:194] Registered new tserver with Master: a1f7afa395cc4d73bc7fa11360a0f26f (127.29.71.130:34493)
I20260929 18:53:58.675529 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 7490
I20260929 18:53:58.675659 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:53:58.675897 29982 external_mini_cluster.cc:1428] Setting key 280bd8713085f1056a88912f3a7a22bf
I20260929 18:53:58.676824 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46251
--builtin_ntp_servers=127.29.71.148:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:53:58.756991 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:53:58.757251 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:53:58.757287 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:53:58.758517 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:53:58.758576 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:53:58.758620 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:53:58.759948 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:53:58.760200 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:53:58.760525 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:53:58.765877 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.765938 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:53:58.765900 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:53:58.766072 7621 server_base.cc:1034] running on GCE node
I20260929 18:53:58.766302 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:53:58.766558 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:53:58.767725 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790708038767714 us; error 44 us; skew 500 ppm
I20260929 18:53:58.768962 7621 webserver.cc:460] Webserver started at http://127.29.71.131:44267/ using document root <none> and password file <none>
I20260929 18:53:58.769201 7621 fs_manager.cc:359] Metadata directory not provided
I20260929 18:53:58.769270 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:53:58.769423 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:53:58.770565 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "497521e415bf4e319bb1ca8255134f18"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "54b2a6da76a59fb9d13e54f47f0a874b"
server_key_iv: "7e5dadc42794ad0cd530e6ac91ae74e2"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.770931 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "497521e415bf4e319bb1ca8255134f18"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "54b2a6da76a59fb9d13e54f47f0a874b"
server_key_iv: "7e5dadc42794ad0cd530e6ac91ae74e2"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.772246 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:53:58.772948 7635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.773116 7621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:53:58.773212 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "497521e415bf4e319bb1ca8255134f18"
format_stamp: "Formatted at 2026-09-29 18:53:58 on dist-test-slave-q2kh"
server_key: "54b2a6da76a59fb9d13e54f47f0a874b"
server_key_iv: "7e5dadc42794ad0cd530e6ac91ae74e2"
server_key_version: "encryptionkey@0"
I20260929 18:53:58.773291 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:53:58.776661 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:53:58.776923 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:53:58.777061 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:53:58.777271 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:53:58.777571 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:53:58.777627 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.777666 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:53:58.777694 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:53:58.783222 7621 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:42397
I20260929 18:53:58.783255 7748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:42397 every 8 connection(s)
I20260929 18:53:58.783574 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:53:58.786477 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.784320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:34877 (local address 127.29.71.190:46251)
0929 18:53:58.784363 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.784365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.784587 (+ 222us) server_negotiation.cc:407] Connection header received
0929 18:53:58.785039 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.785040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.785045 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.785109 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.786271 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:53:58.786273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:53:58.786273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:53:58.786277 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:53:58.786291 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:53:58.786295 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:53:58.786340 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:53:58.786413 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:53:58.786425 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:53:58.786667 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:53:58.784279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46251 (local address 127.29.71.131:34877)
0929 18:53:58.784455 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:53:58.784496 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:53:58.784957 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:53:58.785163 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:53:58.785166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:53:58.785174 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:53:58.786071 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:53:58.786072 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:53:58.786099 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:53:58.786127 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:53:58.786340 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:53:58.786342 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:53:58.786345 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:53:58.786397 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:53:58.786449 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260929 18:53:58.787333 7749 heartbeater.cc:347] Connected to a master server at 127.29.71.190:46251
I20260929 18:53:58.787415 7749 heartbeater.cc:464] Registering TS with master...
I20260929 18:53:58.787523 7749 heartbeater.cc:511] Master 127.29.71.190:46251 requested a full tablet report, sending...
I20260929 18:53:58.787894 7299 ts_manager.cc:194] Registered new tserver with Master: 497521e415bf4e319bb1ca8255134f18 (127.29.71.131:42397)
I20260929 18:53:58.791505 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 7621
I20260929 18:53:58.791603 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:53:58.791802 29982 external_mini_cluster.cc:1428] Setting key 7e988cf05c8fb593fb147ede5520ad61
I20260929 18:53:58.792598 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:53:58.794438 7338 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:53:58.793923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.29.71.190:46251)
0929 18:53:58.793976 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 18:53:58.793978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:53:58.794085 (+ 107us) server_negotiation.cc:407] Connection header received
0929 18:53:58.794141 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:53:58.794142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:53:58.794146 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:53:58.794263 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:53:58.794360 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47120: BlockingRecv error: recv got EOF from 127.0.0.1:47120 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:53:58.794569 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 7359
I20260929 18:53:58.799530 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 7490
I20260929 18:53:58.804111 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 7621
I20260929 18:53:58.808449 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 7268
2026-09-29T18:53:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (532 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:53:58 dist-test-slave-q2kh krb5kdc[7759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:53:58 dist-test-slave-q2kh krb5kdc[7759](info): set up 2 sockets
Sep 29 18:53:58 dist-test-slave-q2kh krb5kdc[7759](info): commencing operation
krb5kdc: starting...
W20260929 18:53:59.894603 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:53:59 dist-test-slave-q2kh krb5kdc[7759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:54:00.017652 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41765
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:41765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:00.100479 7772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:00.100694 7772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:00.100718 7772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:00.101897 7772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:54:00.101940 7772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:00.101955 7772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:00.101967 7772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:54:00.101979 7772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:54:00.104130 7772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:41765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:00.104405 7772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:00.104655 7772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:00.109741 7777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:00.109730 7778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:00.109726 7780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:00.110142 7772 server_base.cc:1034] running on GCE node
I20260929 18:54:00.110476 7772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:00.110797 7772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:00.112008 7772 hybrid_clock.cc:648] HybridClock initialized: now 1790708040111967 us; error 50 us; skew 500 ppm
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:00.120316 7772 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:54:00.120878 7772 webserver.cc:460] Webserver started at http://127.29.71.190:45841/ using document root <none> and password file <none>
I20260929 18:54:00.121150 7772 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:00.121222 7772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:00.121371 7772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:54:00.122535 7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "f26efcd832694a6abcd6eef651be7138"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "2d41fdb449e88686b9b5f075c3a00b27"
server_key_iv: "cdd7b250e7493d0a689b5e54c5ccd3b7"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.122892 7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "f26efcd832694a6abcd6eef651be7138"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "2d41fdb449e88686b9b5f075c3a00b27"
server_key_iv: "cdd7b250e7493d0a689b5e54c5ccd3b7"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.124274 7772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:54:00.125101 7787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.125324 7772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:54:00.125453 7772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "f26efcd832694a6abcd6eef651be7138"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "2d41fdb449e88686b9b5f075c3a00b27"
server_key_iv: "cdd7b250e7493d0a689b5e54c5ccd3b7"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.125530 7772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:00.144330 7772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:00.146754 7772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:00.146934 7772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:00.150959 7772 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:41765
I20260929 18:54:00.151005 7839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:41765 every 8 connection(s)
I20260929 18:54:00.151400 7772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:54:00.152451 7840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:00.152843 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 7772
I20260929 18:54:00.152942 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:54:00.153169 29982 external_mini_cluster.cc:1428] Setting key 076bd79e63c2acac939fda5fe98a210d
I20260929 18:54:00.155794 7840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138: Bootstrap starting.
I20260929 18:54:00.156519 7840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:00.156879 7840 log.cc:824] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:00.157732 7840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138: No bootstrap required, opened a new log
I20260929 18:54:00.159864 7840 raft_consensus.cc:348] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26efcd832694a6abcd6eef651be7138" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41765 } }
I20260929 18:54:00.160073 7840 raft_consensus.cc:374] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:00.160118 7840 raft_consensus.cc:729] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26efcd832694a6abcd6eef651be7138, State: Initialized, Role: FOLLOWER
I20260929 18:54:00.160257 7840 consensus_queue.cc:260] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26efcd832694a6abcd6eef651be7138" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41765 } }
I20260929 18:54:00.160386 7840 raft_consensus.cc:388] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:54:00.160431 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:54:00.160477 7840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:00.161264 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26efcd832694a6abcd6eef651be7138" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41765 } }
I20260929 18:54:00.161422 7840 leader_election.cc:302] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f26efcd832694a6abcd6eef651be7138; no voters:
I20260929 18:54:00.161623 7840 leader_election.cc:288] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:54:00.161669 7845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:54:00.161849 7845 raft_consensus.cc:686] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [term 1 LEADER]: Becoming Leader. State: Replica: f26efcd832694a6abcd6eef651be7138, State: Running, Role: LEADER
I20260929 18:54:00.161921 7840 sys_catalog.cc:564] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:54:00.161948 7845 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26efcd832694a6abcd6eef651be7138" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41765 } }
I20260929 18:54:00.162290 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26efcd832694a6abcd6eef651be7138. Latest consensus state: current_term: 1 leader_uuid: "f26efcd832694a6abcd6eef651be7138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26efcd832694a6abcd6eef651be7138" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41765 } } }
I20260929 18:54:00.162456 7846 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f26efcd832694a6abcd6eef651be7138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26efcd832694a6abcd6eef651be7138" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41765 } } }
I20260929 18:54:00.162500 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [sys.catalog]: This master's current role is: LEADER
I20260929 18:54:00.162545 7846 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138 [sys.catalog]: This master's current role is: LEADER
I20260929 18:54:00.162809 7854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:54:00.163578 7854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:54:00.165630 7854 catalog_manager.cc:1347] Generated new cluster ID: 9a2fd8b35d684db59b5f65b15bd7647d
I20260929 18:54:00.165684 7854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:54:00.165709 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.153779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.29.71.190:41765)
0929 18:54:00.153980 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.153991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.154030 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:54:00.154632 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.154642 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.154707 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.154879 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:00.155479 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.156874 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.157994 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.158478 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.158617 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.161316 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:00.161320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:00.161321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:00.161332 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:00.163756 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.164235 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.164238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.164238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.164362 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.164620 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.164621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.164621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.164886 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:00.165050 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.165347 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.165492 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260929 18:54:00.210310 7854 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:54:00.211261 7854 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:54:00.219411 7854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138: Generated new TSK 0
I20260929 18:54:00.219638 7854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:54:00.251099 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:41765
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:00.333389 7866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:00.333600 7866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:00.333626 7866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:00.334950 7866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:00.335009 7866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:00.335052 7866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:54:00.336585 7866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:41765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:00.336819 7866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:00.337091 7866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:00.342211 7872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:00.342196 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:00.342267 7874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:00.342480 7866 server_base.cc:1034] running on GCE node
I20260929 18:54:00.342648 7866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:00.342834 7866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:00.344015 7866 hybrid_clock.cc:648] HybridClock initialized: now 1790708040344000 us; error 50 us; skew 500 ppm
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:00.346704 7866 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:54:00.347182 7866 webserver.cc:460] Webserver started at http://127.29.71.129:45813/ using document root <none> and password file <none>
I20260929 18:54:00.347425 7866 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:00.347496 7866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:00.347649 7866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:54:00.348740 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "b5d6ee9398e64276870ed94398ccda6e"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "5a1dbb1ebb9ae6f12ae595257537ac25"
server_key_iv: "543df4ce8a3ffbaf209bee99f5038c94"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.349128 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "b5d6ee9398e64276870ed94398ccda6e"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "5a1dbb1ebb9ae6f12ae595257537ac25"
server_key_iv: "543df4ce8a3ffbaf209bee99f5038c94"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.350533 7866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:54:00.351363 7881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.351560 7866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:54:00.351673 7866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "b5d6ee9398e64276870ed94398ccda6e"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "5a1dbb1ebb9ae6f12ae595257537ac25"
server_key_iv: "543df4ce8a3ffbaf209bee99f5038c94"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.351749 7866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:00.371955 7866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:00.373880 7866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:00.374053 7866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:00.374330 7866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:54:00.374655 7866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:54:00.374709 7866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.374745 7866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:54:00.374792 7866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.380438 7866 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:36037
I20260929 18:54:00.380544 7994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:36037 every 8 connection(s)
I20260929 18:54:00.380851 7866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:54:00.386062 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 7866
I20260929 18:54:00.386148 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:54:00.386401 29982 external_mini_cluster.cc:1428] Setting key 7037913491b0ccdb00cfbf0f5f1d860f
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:00.391000 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.381664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:32785 (local address 127.29.71.190:41765)
0929 18:54:00.381735 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.381737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.381943 (+ 206us) server_negotiation.cc:407] Connection header received
0929 18:54:00.382473 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.382474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.382484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.382552 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:00.383491 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.383983 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.384988 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.385191 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.385250 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.387684 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:00.387686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:00.387686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:00.387694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:00.389251 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.389765 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.389767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.389767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.389829 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.390181 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.390182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.390182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.390336 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:00.390410 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.390751 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.390851 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:54:00.391212 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.381657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:41765 (local address 127.29.71.129:32785)
0929 18:54:00.381802 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.381844 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.382366 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.382610 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.382613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.383187 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:54:00.383397 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.383402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.384081 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.384082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.384905 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.384906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.385269 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.385269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.385389 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.385860 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:54:00.385869 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:54:00.387579 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:54:00.389344 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:00.389352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:00.389355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:00.389683 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:00.389974 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:00.389975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:00.389975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:00.390089 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:00.390425 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:54:00.390431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:54:00.390590 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.390709 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.390910 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260929 18:54:00.391686 7995 heartbeater.cc:347] Connected to a master server at 127.29.71.190:41765
I20260929 18:54:00.391808 7995 heartbeater.cc:464] Registering TS with master...
I20260929 18:54:00.391990 7995 heartbeater.cc:511] Master 127.29.71.190:41765 requested a full tablet report, sending...
I20260929 18:54:00.392587 7804 ts_manager.cc:194] Registered new tserver with Master: b5d6ee9398e64276870ed94398ccda6e (127.29.71.129:36037)
I20260929 18:54:00.393543 7804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:32785
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:54:00.413326 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:41765
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:00.493516 8000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:00.493709 8000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:00.493733 8000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:00.494905 8000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:00.494946 8000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:00.494972 8000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:54:00.496495 8000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:41765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:00.496766 8000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:00.497006 8000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:00.502141 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:00.502326 8000 server_base.cc:1034] running on GCE node
W20260929 18:54:00.502148 8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:00.502148 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:00.502614 8000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:00.502868 8000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:00.504048 8000 hybrid_clock.cc:648] HybridClock initialized: now 1790708040504017 us; error 50 us; skew 500 ppm
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:00.506790 8000 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:54:00.507287 8000 webserver.cc:460] Webserver started at http://127.29.71.130:36299/ using document root <none> and password file <none>
I20260929 18:54:00.507594 8000 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:00.507674 8000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:00.507839 8000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:54:00.509038 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "608db8de82834254be9275e3f83b6c64"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "0eded697c5c422ec17b351fcf315c992"
server_key_iv: "b6372af18d7bba5c88a314b71ad22ab9"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.509442 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "608db8de82834254be9275e3f83b6c64"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "0eded697c5c422ec17b351fcf315c992"
server_key_iv: "b6372af18d7bba5c88a314b71ad22ab9"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.510979 8000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:54:00.511780 8015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.511982 8000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:54:00.512096 8000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "608db8de82834254be9275e3f83b6c64"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "0eded697c5c422ec17b351fcf315c992"
server_key_iv: "b6372af18d7bba5c88a314b71ad22ab9"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.512177 8000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:00.528895 8000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:00.531512 8000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:00.531697 8000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:00.531963 8000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:54:00.532307 8000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:54:00.532366 8000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.532407 8000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:54:00.532461 8000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.538944 8000 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:35403
I20260929 18:54:00.538988 8128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:35403 every 8 connection(s)
I20260929 18:54:00.539366 8000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:00.548329 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 8000
I20260929 18:54:00.548455 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:54:00.548648 29982 external_mini_cluster.cc:1428] Setting key 24f4fcbdefee08c63d997bd6d93fe3b8
I20260929 18:54:00.549043 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.540119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:33529 (local address 127.29.71.190:41765)
0929 18:54:00.540166 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.540168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.540368 (+ 200us) server_negotiation.cc:407] Connection header received
0929 18:54:00.540941 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.540942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.540952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.541020 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:00.542042 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.542560 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.543617 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.543804 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.543859 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.546041 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:00.546042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:00.546043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:00.546050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:00.547439 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.547914 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.547915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.547915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.547974 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.548246 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.548247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.548247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.548404 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:00.548485 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.548804 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.548950 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:54:00.549191 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.540053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:41765 (local address 127.29.71.130:33529)
0929 18:54:00.540220 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.540267 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.540820 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.541102 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.541105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.541742 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:54:00.541947 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.541952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.542640 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.542641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.543535 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.543538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.543886 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.543887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.544008 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.544515 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:54:00.544523 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:54:00.545952 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:54:00.547529 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:00.547536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:00.547538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:00.547767 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:00.548053 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:00.548054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:00.548054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:00.548160 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:00.548488 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:54:00.548496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:54:00.548643 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.548766 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.548950 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260929 18:54:00.549665 8129 heartbeater.cc:347] Connected to a master server at 127.29.71.190:41765
I20260929 18:54:00.549780 8129 heartbeater.cc:464] Registering TS with master...
I20260929 18:54:00.549986 8129 heartbeater.cc:511] Master 127.29.71.190:41765 requested a full tablet report, sending...
I20260929 18:54:00.550412 7804 ts_manager.cc:194] Registered new tserver with Master: 608db8de82834254be9275e3f83b6c64 (127.29.71.130:35403)
I20260929 18:54:00.550784 7804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:33529
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:54:00.573419 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:41765
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:00.654192 8134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:00.654454 8134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:00.654497 8134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:00.655709 8134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:00.655777 8134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:00.655822 8134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:54:00.657326 8134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:41765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:00.657606 8134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:00.657850 8134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:00.662776 8142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:00.662838 8139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:00.662906 8140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:00.663161 8134 server_base.cc:1034] running on GCE node
I20260929 18:54:00.663371 8134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:00.663666 8134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:00.664805 8134 hybrid_clock.cc:648] HybridClock initialized: now 1790708040664793 us; error 28 us; skew 500 ppm
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:00.667317 8134 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:54:00.667748 8134 webserver.cc:460] Webserver started at http://127.29.71.131:36753/ using document root <none> and password file <none>
I20260929 18:54:00.667987 8134 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:00.668056 8134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:00.668206 8134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:54:00.669301 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "1476a92cf5604cc1854ef13fc21b9593"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "7d20a749fe0dfe329f1f0492f1909371"
server_key_iv: "9ccb5ad0ae970df6751256d839a927c7"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.669669 8134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "1476a92cf5604cc1854ef13fc21b9593"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "7d20a749fe0dfe329f1f0492f1909371"
server_key_iv: "9ccb5ad0ae970df6751256d839a927c7"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.671128 8134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:54:00.671893 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.672106 8134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:54:00.672225 8134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "1476a92cf5604cc1854ef13fc21b9593"
format_stamp: "Formatted at 2026-09-29 18:54:00 on dist-test-slave-q2kh"
server_key: "7d20a749fe0dfe329f1f0492f1909371"
server_key_iv: "9ccb5ad0ae970df6751256d839a927c7"
server_key_version: "encryptionkey@0"
I20260929 18:54:00.672305 8134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:00.681716 8134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:00.683754 8134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:00.683925 8134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:00.684159 8134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:54:00.684474 8134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:54:00.684528 8134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.684574 8134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:54:00.684600 8134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:00.690318 8134 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:37285
I20260929 18:54:00.690356 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:37285 every 8 connection(s)
I20260929 18:54:00.690687 8134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:54:00 dist-test-slave-q2kh krb5kdc[7759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708040, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:00.698036 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 8134
I20260929 18:54:00.698123 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:54:00.698410 29982 external_mini_cluster.cc:1428] Setting key 570a8d63d427d418b5352eb8dbbab95b
I20260929 18:54:00.700941 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.691504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:33981 (local address 127.29.71.190:41765)
0929 18:54:00.691601 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.691603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.691776 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:54:00.692390 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.692392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.692408 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.692478 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:00.693513 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.694088 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.695160 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.695384 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.695480 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.697894 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:00.697896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:00.697897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:00.697906 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:00.699740 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.700140 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.700141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.700141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.700193 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.700389 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.700389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.700390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.700493 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:00.700544 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.700784 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.700841 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:54:00.701092 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.691422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:41765 (local address 127.29.71.131:33981)
0929 18:54:00.691633 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.691681 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.692248 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.692554 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.692558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.693132 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:54:00.693347 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.693352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.694225 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.694227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.695053 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.695055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.695488 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.695490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.695626 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.696237 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:54:00.696246 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:54:00.697769 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:54:00.699827 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:00.699834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:00.699837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:00.700074 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:00.700242 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:00.700243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:00.700243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:00.700338 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:00.700551 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:54:00.700555 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:54:00.700672 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.700761 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.700886 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260929 18:54:00.701517 8263 heartbeater.cc:347] Connected to a master server at 127.29.71.190:41765
I20260929 18:54:00.701610 8263 heartbeater.cc:464] Registering TS with master...
I20260929 18:54:00.701860 8263 heartbeater.cc:511] Master 127.29.71.190:41765 requested a full tablet report, sending...
I20260929 18:54:00.702284 7804 ts_manager.cc:194] Registered new tserver with Master: 1476a92cf5604cc1854ef13fc21b9593 (127.29.71.131:37285)
I20260929 18:54:00.702657 7804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:33981
I20260929 18:54:00.710683 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:54:00.718454 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.712100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.29.71.190:41765)
0929 18:54:00.712145 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.712147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.712225 (+ 78us) server_negotiation.cc:407] Connection header received
0929 18:54:00.712288 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.712289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.712298 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.712354 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:00.712966 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.713459 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.714273 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.714474 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.714535 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.715912 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:00.715913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:00.715913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:00.715920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:00.717314 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.717672 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.717673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.717673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.717732 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:00.717914 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:00.717914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:00.717915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:00.718062 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:00.718146 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.718307 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.718385 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:54:00.720510 7804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:54:00.722232 7804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:54:00.729022 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.724745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.29.71.131:37285)
0929 18:54:00.724943 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.724950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.725138 (+ 188us) server_negotiation.cc:407] Connection header received
0929 18:54:00.725236 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.725239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.725257 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.725342 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:00.725544 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.726421 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.727798 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.728549 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.728625 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.728685 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.728874 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.728925 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20260929 18:54:00.729045 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.724656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:37285 (local address 127.0.0.1:43002)
0929 18:54:00.725019 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.725064 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.725145 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.725361 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.725365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.725368 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:00.725483 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.725486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.726531 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.726532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.727702 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.727704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.728636 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.728637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.728718 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.728733 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.728836 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.728903 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":66,"threads_started":1}
I20260929 18:54:00.729390 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.725115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.29.71.129:36037)
0929 18:54:00.725300 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.725305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.725477 (+ 172us) server_negotiation.cc:407] Connection header received
0929 18:54:00.725520 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.725523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.725539 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.725606 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:00.725717 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.726907 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.728123 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.728865 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.728978 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.729042 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.729228 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.729300 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260929 18:54:00.729323 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.724864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36037 (local address 127.0.0.1:39382)
0929 18:54:00.725361 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.725370 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.725418 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.725611 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.725612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.725614 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:00.725661 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.725662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.727012 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.727013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.728046 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.728048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.729075 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.729076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.729154 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.729158 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.729214 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.729256 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":101,"threads_started":1}
I20260929 18:54:00.730185 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.724660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:35403 (local address 127.0.0.1:58954)
0929 18:54:00.725065 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.725075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.725154 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.725422 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.725423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.725425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:00.725476 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.725478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.728116 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.728117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.728916 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.728918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.729924 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.729925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.729999 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.730003 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.730060 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.730097 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":87,"threads_started":1}
I20260929 18:54:00.730396 8197 tablet_service.cc:1467] Processing CreateTablet for tablet f3f215701ba34ba0b84005ddf2ad9a95 (DEFAULT_TABLE table=test-table [id=652fe6fbac32459c8f0a9703a6965ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:54:00.730341 7929 tablet_service.cc:1467] Processing CreateTablet for tablet f3f215701ba34ba0b84005ddf2ad9a95 (DEFAULT_TABLE table=test-table [id=652fe6fbac32459c8f0a9703a6965ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:54:00.730563 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.724825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.29.71.130:35403)
0929 18:54:00.725029 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.725034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.725233 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:54:00.725270 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.725272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.725288 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.725353 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:00.725785 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.726899 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.729001 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.729763 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.729834 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.729895 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.730090 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.730137 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":150,"threads_started":1}
I20260929 18:54:00.730701 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f215701ba34ba0b84005ddf2ad9a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:00.730705 7929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f215701ba34ba0b84005ddf2ad9a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:00.731206 8063 tablet_service.cc:1467] Processing CreateTablet for tablet f3f215701ba34ba0b84005ddf2ad9a95 (DEFAULT_TABLE table=test-table [id=652fe6fbac32459c8f0a9703a6965ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:54:00.731485 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f215701ba34ba0b84005ddf2ad9a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:00.733137 8280 tablet_bootstrap.cc:492] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Bootstrap starting.
I20260929 18:54:00.733790 8280 tablet_bootstrap.cc:654] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:00.734196 8281 tablet_bootstrap.cc:492] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: Bootstrap starting.
I20260929 18:54:00.734477 8280 log.cc:824] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:00.734858 8281 tablet_bootstrap.cc:654] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:00.735201 8280 tablet_bootstrap.cc:492] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: No bootstrap required, opened a new log
I20260929 18:54:00.735282 8280 ts_tablet_manager.cc:1397] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:54:00.735327 8281 log.cc:824] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:00.735677 8282 tablet_bootstrap.cc:492] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Bootstrap starting.
I20260929 18:54:00.736007 8281 tablet_bootstrap.cc:492] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: No bootstrap required, opened a new log
I20260929 18:54:00.736057 8281 ts_tablet_manager.cc:1397] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:54:00.736748 8280 raft_consensus.cc:348] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.736907 8280 raft_consensus.cc:374] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:00.736968 8280 raft_consensus.cc:729] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1476a92cf5604cc1854ef13fc21b9593, State: Initialized, Role: FOLLOWER
I20260929 18:54:00.737119 8280 consensus_queue.cc:260] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.737389 8263 heartbeater.cc:503] Master 127.29.71.190:41765 was elected leader, sending a full tablet report...
I20260929 18:54:00.737654 8280 ts_tablet_manager.cc:1428] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:54:00.737634 8281 raft_consensus.cc:348] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.737785 8281 raft_consensus.cc:374] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:00.737820 8281 raft_consensus.cc:729] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d6ee9398e64276870ed94398ccda6e, State: Initialized, Role: FOLLOWER
I20260929 18:54:00.737931 8281 consensus_queue.cc:260] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.738231 8281 ts_tablet_manager.cc:1428] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:54:00.738366 7995 heartbeater.cc:503] Master 127.29.71.190:41765 was elected leader, sending a full tablet report...
I20260929 18:54:00.738933 8282 tablet_bootstrap.cc:654] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:00.739372 8282 log.cc:824] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:00.739957 8282 tablet_bootstrap.cc:492] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: No bootstrap required, opened a new log
I20260929 18:54:00.740036 8282 ts_tablet_manager.cc:1397] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:54:00.741346 8282 raft_consensus.cc:348] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.741467 8282 raft_consensus.cc:374] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:00.741488 8282 raft_consensus.cc:729] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608db8de82834254be9275e3f83b6c64, State: Initialized, Role: FOLLOWER
I20260929 18:54:00.741596 8282 consensus_queue.cc:260] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.741855 8282 ts_tablet_manager.cc:1428] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:54:00.741873 8129 heartbeater.cc:503] Master 127.29.71.190:41765 was elected leader, sending a full tablet report...
W20260929 18:54:00.789991 8130 tablet.cc:2369] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:54:00.881628 7996 tablet.cc:2369] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:54:00.901326 8285 raft_consensus.cc:482] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:54:00.901489 8285 raft_consensus.cc:504] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.901893 8285 leader_election.cc:288] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608db8de82834254be9275e3f83b6c64 (127.29.71.130:35403), b5d6ee9398e64276870ed94398ccda6e (127.29.71.129:36037)
I20260929 18:54:00.904889 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.902110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:35639 (local address 127.29.71.129:36037)
0929 18:54:00.902176 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.902178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.902272 (+ 94us) server_negotiation.cc:407] Connection header received
0929 18:54:00.902313 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.902315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.902328 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.902404 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:00.902583 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.903067 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.904156 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.904523 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.904600 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.904618 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.904772 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.904808 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:54:00.904881 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.902195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:38369 (local address 127.29.71.130:35403)
0929 18:54:00.902244 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.902246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.902400 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:54:00.902428 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.902429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.902441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.902493 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:00.902664 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.903137 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.904165 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.904526 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.904615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.904623 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.904771 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.904805 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:54:00.904907 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.902089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:35403 (local address 127.29.71.131:38369)
0929 18:54:00.902269 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.902280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.902347 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.902507 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.902508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.902509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:00.902589 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.902591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.903225 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.903226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.903964 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.903966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.904627 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.904628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.904695 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.904698 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.904763 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.904804 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":132,"threads_started":1}
I20260929 18:54:00.904915 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.902071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36037 (local address 127.29.71.131:35639)
0929 18:54:00.902143 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.902176 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:00.902227 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:00.902426 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:00.902426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:00.902428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:00.902501 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.902503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.903147 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.903148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.903962 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:00.903964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.904605 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.904607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:00.904686 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.904689 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:54:00.904767 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:54:00.904806 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":18}
I20260929 18:54:00.905246 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f215701ba34ba0b84005ddf2ad9a95" candidate_uuid: "1476a92cf5604cc1854ef13fc21b9593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608db8de82834254be9275e3f83b6c64" is_pre_election: true
I20260929 18:54:00.905371 8083 raft_consensus.cc:2454] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1476a92cf5604cc1854ef13fc21b9593 in term 0.
I20260929 18:54:00.905609 8150 leader_election.cc:302] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1476a92cf5604cc1854ef13fc21b9593, 608db8de82834254be9275e3f83b6c64; no voters:
I20260929 18:54:00.905589 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f215701ba34ba0b84005ddf2ad9a95" candidate_uuid: "1476a92cf5604cc1854ef13fc21b9593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d6ee9398e64276870ed94398ccda6e" is_pre_election: true
I20260929 18:54:00.905755 7949 raft_consensus.cc:2454] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1476a92cf5604cc1854ef13fc21b9593 in term 0.
I20260929 18:54:00.905762 8285 raft_consensus.cc:2781] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:54:00.905821 8285 raft_consensus.cc:482] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:54:00.905879 8285 raft_consensus.cc:3037] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:54:00.906647 8285 raft_consensus.cc:504] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.906867 8285 leader_election.cc:288] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [CANDIDATE]: Term 1 election: Requested vote from peers 608db8de82834254be9275e3f83b6c64 (127.29.71.130:35403), b5d6ee9398e64276870ed94398ccda6e (127.29.71.129:36037)
I20260929 18:54:00.906952 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f215701ba34ba0b84005ddf2ad9a95" candidate_uuid: "1476a92cf5604cc1854ef13fc21b9593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608db8de82834254be9275e3f83b6c64"
I20260929 18:54:00.907006 7949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f215701ba34ba0b84005ddf2ad9a95" candidate_uuid: "1476a92cf5604cc1854ef13fc21b9593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5d6ee9398e64276870ed94398ccda6e"
I20260929 18:54:00.907032 8083 raft_consensus.cc:3037] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:54:00.907095 7949 raft_consensus.cc:3037] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:54:00.907843 8083 raft_consensus.cc:2454] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1476a92cf5604cc1854ef13fc21b9593 in term 1.
I20260929 18:54:00.907859 7949 raft_consensus.cc:2454] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1476a92cf5604cc1854ef13fc21b9593 in term 1.
I20260929 18:54:00.907994 8150 leader_election.cc:302] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1476a92cf5604cc1854ef13fc21b9593, 608db8de82834254be9275e3f83b6c64; no voters:
I20260929 18:54:00.908120 8285 raft_consensus.cc:2781] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:54:00.908258 8285 raft_consensus.cc:686] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 1 LEADER]: Becoming Leader. State: Replica: 1476a92cf5604cc1854ef13fc21b9593, State: Running, Role: LEADER
I20260929 18:54:00.908367 8285 consensus_queue.cc:237] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } }
I20260929 18:54:00.909315 7803 catalog_manager.cc:5495] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1476a92cf5604cc1854ef13fc21b9593 (127.29.71.131). New cstate: current_term: 1 leader_uuid: "1476a92cf5604cc1854ef13fc21b9593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1476a92cf5604cc1854ef13fc21b9593" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 } health_report { overall_health: UNKNOWN } } }
W20260929 18:54:00.941258 8264 tablet.cc:2369] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:54:00.958081 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.955540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.29.71.131:37285)
0929 18:54:00.955608 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:00.955610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:00.955651 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:54:00.955738 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:00.955739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:00.955762 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:00.955851 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:54:00.956032 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.956507 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.957274 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:00.957486 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:00.957542 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:00.957702 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:54:00.957839 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:54:00.957930 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:00.957990 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:54:00.958008 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:54:00.959769 8083 raft_consensus.cc:1261] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 1 FOLLOWER]: Refusing update from remote peer 1476a92cf5604cc1854ef13fc21b9593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:54:00.959780 7949 raft_consensus.cc:1261] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e [term 1 FOLLOWER]: Refusing update from remote peer 1476a92cf5604cc1854ef13fc21b9593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:54:00.960160 8290 consensus_queue.cc:1035] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d6ee9398e64276870ed94398ccda6e" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:54:00.960273 8285 consensus_queue.cc:1035] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608db8de82834254be9275e3f83b6c64" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:54:00.963070 8294 mvcc.cc:205] Tried to move back new op lower bound from 7334740135768662016 to 7334740135561433088. Current Snapshot: MvccSnapshot[applied={T|T < 7334740135768662016}]
I20260929 18:54:00.963124 8293 mvcc.cc:205] Tried to move back new op lower bound from 7334740135768662016 to 7334740135561433088. Current Snapshot: MvccSnapshot[applied={T|T < 7334740135768662016 or (T in {7334740135768662016})}]
I20260929 18:54:00.964675 7804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:54:00.964767 7804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:54:00.965067 8295 mvcc.cc:205] Tried to move back new op lower bound from 7334740135768662016 to 7334740135561433088. Current Snapshot: MvccSnapshot[applied={T|T < 7334740135768662016 or (T in {7334740135768662016})}]
I20260929 18:54:00.965492 7804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f26efcd832694a6abcd6eef651be7138: Sending DeleteTablet for 3 replicas of tablet f3f215701ba34ba0b84005ddf2ad9a95
I20260929 18:54:00.965829 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f215701ba34ba0b84005ddf2ad9a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:54:00 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:58954
I20260929 18:54:00.965859 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f215701ba34ba0b84005ddf2ad9a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:54:00 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:43002
I20260929 18:54:00.965942 7929 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f215701ba34ba0b84005ddf2ad9a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:54:00 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:39382
I20260929 18:54:00.966138 8304 tablet_replica.cc:330] T f3f215701ba34ba0b84005ddf2ad9a95 P b5d6ee9398e64276870ed94398ccda6e: stopping tablet replica
I20260929 18:54:00.966194 8303 tablet_replica.cc:330] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: stopping tablet replica
I20260929 18:54:00.966270 8302 tablet_replica.cc:330] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: stopping tablet replica
I20260929 18:54:00.966320 8303 raft_consensus.cc:2229] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:54:00.966418 8302 raft_consensus.cc:2229] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:54:00.966447 8303 raft_consensus.cc:2258] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:54:00.966495 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 7866
I20260929 18:54:00.968137 8303 ts_tablet_manager.cc:1905] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:54:00.966557 8302 raft_consensus.cc:2258] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:54:00.969094 8302 ts_tablet_manager.cc:1905] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:54:00.970124 8303 ts_tablet_manager.cc:1918] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:54:00.970265 8303 log.cc:1196] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/f3f215701ba34ba0b84005ddf2ad9a95
I20260929 18:54:00.970666 8303 ts_tablet_manager.cc:1939] T f3f215701ba34ba0b84005ddf2ad9a95 P 1476a92cf5604cc1854ef13fc21b9593: Deleting consensus metadata
I20260929 18:54:00.970844 8302 ts_tablet_manager.cc:1918] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:54:00.970953 8302 log.cc:1196] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/f3f215701ba34ba0b84005ddf2ad9a95
I20260929 18:54:00.971169 7790 catalog_manager.cc:4841] TS 1476a92cf5604cc1854ef13fc21b9593 (127.29.71.131:37285): tablet f3f215701ba34ba0b84005ddf2ad9a95 (table test-table [id=652fe6fbac32459c8f0a9703a6965ee6]) successfully deleted
I20260929 18:54:00.971264 8302 ts_tablet_manager.cc:1939] T f3f215701ba34ba0b84005ddf2ad9a95 P 608db8de82834254be9275e3f83b6c64: Deleting consensus metadata
I20260929 18:54:00.971784 7788 catalog_manager.cc:4841] TS 608db8de82834254be9275e3f83b6c64 (127.29.71.130:35403): tablet f3f215701ba34ba0b84005ddf2ad9a95 (table test-table [id=652fe6fbac32459c8f0a9703a6965ee6]) successfully deleted
W20260929 18:54:00.972064 7790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:36037 (error 108)
I20260929 18:54:00.972281 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 8000
I20260929 18:54:00.972856 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:00.972402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36037 (local address 127.0.0.1:39394)
0929 18:54:00.972763 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:00.972808 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:36037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341}
W20260929 18:54:00.978292 7790 catalog_manager.cc:4584] TS b5d6ee9398e64276870ed94398ccda6e (127.29.71.129:36037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3f215701ba34ba0b84005ddf2ad9a95: Network error: Client connection negotiation failed: client connection to 127.29.71.129:36037: connect: Connection refused (error 111)
I20260929 18:54:00.979797 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 8134
I20260929 18:54:00.984455 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 7772
2026-09-29T18:54:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2175 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:54:01 dist-test-slave-q2kh krb5kdc[8308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:54:01 dist-test-slave-q2kh krb5kdc[8308](info): set up 2 sockets
Sep 29 18:54:01 dist-test-slave-q2kh krb5kdc[8308](info): commencing operation
krb5kdc: starting...
W20260929 18:54:02.022482 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:54:02 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:54:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:54:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/rangeradmin_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/rangeradmin_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/rangerlookup_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/rangerlookup_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
I20260929 18:54:02.193048 29982 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWZtMeW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/postgres -l logfile start
2026-09-29 18:54:04.838 UTC [8338] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:54:04.838 UTC [8338] LOG: listening on IPv4 address "127.29.71.148", port 51323
2026-09-29 18:54:04.841 UTC [8338] LOG: listening on Unix socket "/tmp/.s.PGSQL.51323"
2026-09-29 18:54:04.877 UTC [8341] LOG: database system was shut down at 2026-09-29 18:54:03 UTC
2026-09-29 18:54:04.880 UTC [8338] LOG: database system is ready to accept connections
I20260929 18:54:05.830250 29982 mini_postgres.cc:96] Postgres bound to 51323
2026-09-29 18:54:05.836 UTC [8349] FATAL: database "slave" does not exist
127.29.71.148:51323 - accepting connections
I20260929 18:54:05.836704 29982 mini_ranger.cc:161] Starting Ranger...
I20260929 18:54:05.851423 29982 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:54:06.358719 29982 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:54:06.358785 29982 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-admin
2026-09-29 18:54:06,642 [I] DB FLAVOR :POSTGRES
2026-09-29 18:54:06,643 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:54:06,643 [I] Checking connection..
2026-09-29 18:54:06,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:54:06,902 [I] Checking connection passed.
2026-09-29 18:54:06,902 [I] --------- Verifying version history table ---------
2026-09-29 18:54:06,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:54:07,152 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:54:07,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:54:07,385 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:54:07,386 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:54:07,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:54:07.650 UTC [8413] WARNING: there is no transaction in progress
2026-09-29 18:54:07,656 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:54:07,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:54:07,939 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:54:07,939 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:54:07,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:54:08,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:54:08,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q2kh.c.gcp-upstream.internal','N') ;"
2026-09-29 18:54:08,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:54:08,943 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:54:08,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:54:09,186 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:54:09,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:54:09,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:54:09,714 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:54:09,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:54:10.759 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.776 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.792 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.809 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.828 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.846 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.903 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.911 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.926 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.937 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.947 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.962 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.971 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.979 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.985 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:10.993 UTC [8521] WARNING: there is no transaction in progress
2026-09-29 18:54:11,681 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:54:11,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:54:11,932 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:54:11,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:54:12,180 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:54:12,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:54:12,422 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:54:12,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:54:12,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:54:12,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q2kh.c.gcp-upstream.internal';"
2026-09-29 18:54:13,152 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:54:13,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:54:13,402 [I] DB_PATCHES have already been applied
I20260929 18:54:13.408288 29982 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWZtMeW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:54:13.408393 29982 mini_ranger.cc:193] Using host: 127.29.71.148
I20260929 18:54:13.411327 29982 mini_ranger.cc:227] Ranger admin URL: http://127.29.71.148:53961
Sep 29, 2026 6:54:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:54:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:54:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:54:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:54:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:54:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:54:14.541393 29982 mini_ranger.cc:160] Time spent starting Ranger: real 8.705s user 0.000s sys 0.009s
Sep 29, 2026 6:54:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:54:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:54:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:54:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:54:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:54:35.589025 29982 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/rangerkms_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/rangerkms_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.71.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.71.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:54:35 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708075, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:54:35.676122 29982 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:54:35.685884 29982 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:54:36.638576 29982 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:54:36.638666 29982 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms
I20260929 18:54:36.638685 29982 mini_ranger_kms.cc:227] Using postgres at 127.29.71.148:51323
2026-09-29 18:54:36,820 [I] DB FLAVOR :POSTGRES
2026-09-29 18:54:36,821 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:54:36,821 [I] Checking connection
2026-09-29 18:54:36,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:54:37,086 [I] connection success
2026-09-29 18:54:37,087 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:54:37,087 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:54:37,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:54:37,411 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:54:37,411 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:54:37,411 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:54:37,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:54:37,707 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:54:38.226065 29982 mini_ranger_kms.cc:310] Created kms service
I20260929 18:54:38.418881 29982 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:54:38.461092 29982 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:54:38 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708075, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:54:38.856805 29982 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:54:38.856928 29982 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:54:38.856952 29982 mini_ranger_kms.cc:240] Using host: 127.29.71.148
I20260929 18:54:38.861318 29982 mini_ranger_kms.cc:276] Ranger KMS PID: 8701
I20260929 18:54:38.861459 29982 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.71.148:53831
Sep 29, 2026 6:54:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:54:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:54:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:54:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:54:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:54:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:54:41.226245 29982 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.550s user 0.000s sys 0.165s
I20260929 18:54:41.226365 29982 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:54:41.226441 29982 mini_ranger_kms.cc:401] 127.29.71.148:53831/kms/v1/keys
Sep 29 18:54:41 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708081, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.71.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:54:41 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708081, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.71.148@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
Sep 29 18:54:42 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708081, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.71.148@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
Sep 29 18:54:44 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:54:44.614135 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34089
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:34089
--ranger_config_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:53831/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:44.708179 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:44.708397 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:44.708420 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:44.709563 8752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:54:44.709606 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:44.709621 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:44.709633 8752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:54:44.709646 8752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:54:44.711241 8752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.71.148:53831/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:34089
--ranger_config_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:44.711470 8752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:44.711714 8752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:44.717494 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:44.717505 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:44.717841 8752 server_base.cc:1034] running on GCE node
W20260929 18:54:44.717558 8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:44.718127 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:44.718482 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:44.719659 8752 hybrid_clock.cc:648] HybridClock initialized: now 1790708084719642 us; error 30 us; skew 500 ppm
Sep 29 18:54:44 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708084, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:44.722675 8752 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:54:44.723152 8752 webserver.cc:460] Webserver started at http://127.29.71.190:44475/ using document root <none> and password file <none>
I20260929 18:54:44.723397 8752 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:44.723455 8752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:44.723613 8752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:54:44 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708084, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:54:44.781586 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "dc060fd7713c4d47ad3087d63a5a5119"
format_stamp: "Formatted at 2026-09-29 18:54:44 on dist-test-slave-q2kh"
server_key: "09ea780576ddfa6be7f503b09b6b2dd2"
server_key_iv: "fb5343b3986be800da9d9877982ba79e"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:44.781937 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "dc060fd7713c4d47ad3087d63a5a5119"
format_stamp: "Formatted at 2026-09-29 18:54:44 on dist-test-slave-q2kh"
server_key: "09ea780576ddfa6be7f503b09b6b2dd2"
server_key_iv: "fb5343b3986be800da9d9877982ba79e"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:44.783391 8752 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:54:44.797778 8767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:44.798125 8752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:54:44.798256 8752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "dc060fd7713c4d47ad3087d63a5a5119"
format_stamp: "Formatted at 2026-09-29 18:54:44 on dist-test-slave-q2kh"
server_key: "09ea780576ddfa6be7f503b09b6b2dd2"
server_key_iv: "fb5343b3986be800da9d9877982ba79e"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:44.798452 8752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:44.820238 8752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:44.822407 8752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:44.822562 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:44.827977 8752 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:34089
I20260929 18:54:44.827999 8819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:34089 every 8 connection(s)
I20260929 18:54:44.828414 8752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:54:44.829082 8820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:44.830900 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 8752
I20260929 18:54:44.831012 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:54:44.831203 29982 external_mini_cluster.cc:1428] Setting key 23c0522f5cf7d041cddf299ab14107f8
I20260929 18:54:44.831442 8820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119: Bootstrap starting.
I20260929 18:54:44.832181 8820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:44.832981 8820 log.cc:824] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:44.833726 8820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119: No bootstrap required, opened a new log
I20260929 18:54:44.835820 8820 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fd7713c4d47ad3087d63a5a5119" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34089 } }
I20260929 18:54:44.836042 8820 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:44.836095 8820 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc060fd7713c4d47ad3087d63a5a5119, State: Initialized, Role: FOLLOWER
I20260929 18:54:44.836262 8820 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fd7713c4d47ad3087d63a5a5119" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34089 } }
I20260929 18:54:44.836352 8820 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:54:44.836406 8820 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:54:44.836462 8820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:54:44.837122 8820 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fd7713c4d47ad3087d63a5a5119" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34089 } }
I20260929 18:54:44.837242 8820 leader_election.cc:302] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc060fd7713c4d47ad3087d63a5a5119; no voters:
I20260929 18:54:44.837401 8820 leader_election.cc:288] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:54:44.837443 8825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:54:44.837620 8825 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [term 1 LEADER]: Becoming Leader. State: Replica: dc060fd7713c4d47ad3087d63a5a5119, State: Running, Role: LEADER
I20260929 18:54:44.837710 8820 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:54:44.837754 8825 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fd7713c4d47ad3087d63a5a5119" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34089 } }
I20260929 18:54:44.838146 8825 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc060fd7713c4d47ad3087d63a5a5119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fd7713c4d47ad3087d63a5a5119" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34089 } } }
I20260929 18:54:44.838301 8825 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [sys.catalog]: This master's current role is: LEADER
I20260929 18:54:44.838457 8825 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc060fd7713c4d47ad3087d63a5a5119. Latest consensus state: current_term: 1 leader_uuid: "dc060fd7713c4d47ad3087d63a5a5119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc060fd7713c4d47ad3087d63a5a5119" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34089 } } }
I20260929 18:54:44.838546 8825 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119 [sys.catalog]: This master's current role is: LEADER
I20260929 18:54:44.838681 8832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:54:44.839155 8832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:54:44 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:44.840104 8820 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:54:44.846232 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:44.831707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.29.71.190:34089)
0929 18:54:44.831873 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:44.831892 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:44.831923 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:54:44.832448 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:44.832456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:44.832517 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:44.832666 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:44.833582 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:44.834948 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:44.836014 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:44.836618 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:44.836723 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:44.839814 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:44.839818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:44.839819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:44.839829 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:44.844243 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:44.844721 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:44.844724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:44.844725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:44.844866 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:44.845150 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:44.845151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:44.845151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:44.845464 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:44.845656 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:44.845839 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:54:44.845987 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260929 18:54:44.847709 8832 catalog_manager.cc:1347] Generated new cluster ID: edc832e70a8340248f473086b31b3be9
I20260929 18:54:44.847816 8832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:54:44.855250 8832 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:54:44.856030 8832 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:54:44.868148 8832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119: Generated new TSK 0
I20260929 18:54:44.868443 8832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:54:45.572777 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:53831/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34089
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:45.659739 8865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:45.659938 8865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:45.659961 8865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:45.661088 8865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:45.661128 8865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:45.661154 8865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:54:45.662729 8865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.71.148:53831/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:45.662954 8865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:45.663204 8865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:45.668255 8870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:45.668385 8871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:45.668244 8873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:45.669070 8865 server_base.cc:1034] running on GCE node
I20260929 18:54:45.669291 8865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:45.669548 8865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:45.670714 8865 hybrid_clock.cc:648] HybridClock initialized: now 1790708085670700 us; error 32 us; skew 500 ppm
Sep 29 18:54:45 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:45.676687 8865 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:54:45.677142 8865 webserver.cc:460] Webserver started at http://127.29.71.129:35843/ using document root <none> and password file <none>
I20260929 18:54:45.677423 8865 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:45.677491 8865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:45.677639 8865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:54:45 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:54:45.694360 8865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "91bd5efe1124462b8ab042ab137f4549"
format_stamp: "Formatted at 2026-09-29 18:54:45 on dist-test-slave-q2kh"
server_key: "ad326c3f4f2b2e149710d647573f2624"
server_key_iv: "21103616a6e5231ed38d3629af145ba0"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:45.694921 8865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "91bd5efe1124462b8ab042ab137f4549"
format_stamp: "Formatted at 2026-09-29 18:54:45 on dist-test-slave-q2kh"
server_key: "ad326c3f4f2b2e149710d647573f2624"
server_key_iv: "21103616a6e5231ed38d3629af145ba0"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:45.696602 8865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:54:45.710353 8880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:45.710603 8865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:54:45.710721 8865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "91bd5efe1124462b8ab042ab137f4549"
format_stamp: "Formatted at 2026-09-29 18:54:45 on dist-test-slave-q2kh"
server_key: "ad326c3f4f2b2e149710d647573f2624"
server_key_iv: "21103616a6e5231ed38d3629af145ba0"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:45.710819 8865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:45.723551 8865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:45.725744 8865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:45.725901 8865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:45.726158 8865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:54:45.726544 8865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:54:45.726594 8865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:45.726660 8865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:54:45.726702 8865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:45.735289 8865 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37699
I20260929 18:54:45.735384 8995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37699 every 8 connection(s)
I20260929 18:54:45.735970 8865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:54:45.738647 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 8865
I20260929 18:54:45.738775 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:54:45.739003 29982 external_mini_cluster.cc:1428] Setting key 871846156501043ebd3afc6d7d150c0e
Sep 29 18:54:45 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:45.748638 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:45.736792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34089 (local address 127.29.71.129:44865)
0929 18:54:45.737334 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:45.737395 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:45.738480 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:45.738997 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:45.739002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:45.739251 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:54:45.739547 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:45.739553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.740652 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.740653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:45.741954 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:45.741957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.742496 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.742497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:45.742639 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:45.743226 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:54:45.743237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:54:45.744895 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:54:45.747035 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:45.747042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:45.747044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:45.747322 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:45.747551 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:45.747552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:45.747552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:45.747678 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:45.747985 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:54:45.747991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:54:45.748154 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 18:54:45.748242 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:54:45.748402 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":150,"threads_started":1}
I20260929 18:54:45.749421 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:45.737199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:44865 (local address 127.29.71.190:34089)
0929 18:54:45.738669 (+ 1470us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:45.738671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:45.738679 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:54:45.738704 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:45.738705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:45.738778 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:45.739694 (+ 916us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:45.739734 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.740543 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.742055 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.742399 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.745058 (+ 2659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:45.745108 (+ 50us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:45.745109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:45.745109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:45.745114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:45.746938 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:45.747419 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:45.747420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:45.747420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:45.747481 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:45.747775 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:45.747775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:45.747776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:45.747909 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:45.749182 (+ 1273us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:45.749226 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 18:54:45.749313 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1440,"thread_start_us":84,"threads_started":1}
I20260929 18:54:45.749768 8996 heartbeater.cc:347] Connected to a master server at 127.29.71.190:34089
I20260929 18:54:45.749881 8996 heartbeater.cc:464] Registering TS with master...
I20260929 18:54:45.750082 8996 heartbeater.cc:511] Master 127.29.71.190:34089 requested a full tablet report, sending...
I20260929 18:54:45.750705 8784 ts_manager.cc:194] Registered new tserver with Master: 91bd5efe1124462b8ab042ab137f4549 (127.29.71.129:37699)
I20260929 18:54:45.751590 8784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:44865
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:54:45.777280 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:53831/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34089
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:45.886035 9002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:45.886320 9002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:45.886374 9002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:45.887569 9002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:45.887650 9002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:45.887696 9002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:54:45.889475 9002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.71.148:53831/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:45.889737 9002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:45.889992 9002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:45.895030 9012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:45.895037 9010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:45.895037 9009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:45.895758 9002 server_base.cc:1034] running on GCE node
I20260929 18:54:45.896095 9002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:45.896377 9002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:45.897541 9002 hybrid_clock.cc:648] HybridClock initialized: now 1790708085897515 us; error 41 us; skew 500 ppm
Sep 29 18:54:45 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:45.900629 9002 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:54:45.901136 9002 webserver.cc:460] Webserver started at http://127.29.71.130:45111/ using document root <none> and password file <none>
I20260929 18:54:45.901378 9002 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:45.901443 9002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:45.901580 9002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:54:45 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
Sep 29 18:54:45 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:45.915705 9002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "4825eca72b05415a9ded842d29a6304a"
format_stamp: "Formatted at 2026-09-29 18:54:45 on dist-test-slave-q2kh"
server_key: "4464dae0a53ddba0fe238fa6efe237da"
server_key_iv: "2042a2a8f941108ab099ff33a2db3024"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:45.916273 9002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "4825eca72b05415a9ded842d29a6304a"
format_stamp: "Formatted at 2026-09-29 18:54:45 on dist-test-slave-q2kh"
server_key: "4464dae0a53ddba0fe238fa6efe237da"
server_key_iv: "2042a2a8f941108ab099ff33a2db3024"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:45.928733 9002 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.001s sys 0.000s
I20260929 18:54:45.950299 9019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:45.950606 9002 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260929 18:54:45.950718 9002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "4825eca72b05415a9ded842d29a6304a"
format_stamp: "Formatted at 2026-09-29 18:54:45 on dist-test-slave-q2kh"
server_key: "4464dae0a53ddba0fe238fa6efe237da"
server_key_iv: "2042a2a8f941108ab099ff33a2db3024"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:45.950831 9002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:45.964447 9002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:45.966794 9002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:45.966998 9002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:45.967291 9002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:54:45.967674 9002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:54:45.967754 9002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:45.967823 9002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:54:45.967876 9002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:45.975909 9002 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:36019
I20260929 18:54:45.975970 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:36019 every 8 connection(s)
I20260929 18:54:45.976410 9002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:54:45 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:45.984567 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 9002
I20260929 18:54:45.984716 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:54:45.984918 29982 external_mini_cluster.cc:1428] Setting key 6e4ef0ca8f17f18ad409a58cc5c81df0
I20260929 18:54:45.987834 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:45.977155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:52873 (local address 127.29.71.190:34089)
0929 18:54:45.977220 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:45.977222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:45.977470 (+ 248us) server_negotiation.cc:407] Connection header received
0929 18:54:45.978094 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:45.978096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:45.978111 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:45.978218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:45.978944 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.979708 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.981188 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.981498 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.981545 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:45.984167 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:45.984169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:45.984169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:45.984177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:45.986096 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:45.986636 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:45.986638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:45.986638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:45.986701 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:45.987021 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:45.987023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:45.987023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:45.987165 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:45.987210 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:45.987643 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 18:54:45.987738 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:54:45.988592 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:45.977141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34089 (local address 127.29.71.130:52873)
0929 18:54:45.977300 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:45.977356 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:45.977980 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:45.978298 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:45.978303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:45.978550 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:54:45.978831 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:45.978838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.979818 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.979819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:45.981084 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:45.981086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:45.981635 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:45.981636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:45.981774 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:45.982418 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:54:45.982428 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:54:45.984066 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:54:45.986220 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:45.986229 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:45.986231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:45.986554 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:45.986779 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:45.986780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:45.986780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:45.986927 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:45.987261 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:54:45.987267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:54:45.987465 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 18:54:45.988144 (+ 679us) client_negotiation.cc:240] Negotiation successful
0929 18:54:45.988344 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260929 18:54:45.990638 9133 heartbeater.cc:347] Connected to a master server at 127.29.71.190:34089
I20260929 18:54:45.990747 9133 heartbeater.cc:464] Registering TS with master...
I20260929 18:54:45.990955 9133 heartbeater.cc:511] Master 127.29.71.190:34089 requested a full tablet report, sending...
I20260929 18:54:45.991400 8784 ts_manager.cc:194] Registered new tserver with Master: 4825eca72b05415a9ded842d29a6304a (127.29.71.130:36019)
I20260929 18:54:45.991925 8784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:52873
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:54:46.030565 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:53831/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34089
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:54:46.158726 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:54:46.159090 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:54:46.159189 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:54:46.160923 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:54:46.161152 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:54:46.161247 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:54:46.163213 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.71.148:53831/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:34089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:54:46.163492 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:54:46.163728 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:54:46.170809 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:46.171074 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:54:46.171253 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:54:46.172325 9139 server_base.cc:1034] running on GCE node
I20260929 18:54:46.172542 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:54:46.172802 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:54:46.176724 9139 hybrid_clock.cc:648] HybridClock initialized: now 1790708086176700 us; error 40 us; skew 500 ppm
Sep 29 18:54:46 dist-test-slave-q2kh krb5kdc[8308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708086, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:54:46.179908 9139 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:54:46.180413 9139 webserver.cc:460] Webserver started at http://127.29.71.131:43389/ using document root <none> and password file <none>
I20260929 18:54:46.180698 9139 fs_manager.cc:359] Metadata directory not provided
I20260929 18:54:46.184856 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:54:46.185025 9139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:54:46 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708086, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:54:46.202405 9139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "b7edf00c4cdd4ab4a1f256aa650196ab"
format_stamp: "Formatted at 2026-09-29 18:54:46 on dist-test-slave-q2kh"
server_key: "8ae2aba14f8bb0ea52314e0f4de04f7c"
server_key_iv: "5ecad473be14603c896ed814d24f904c"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:46.202950 9139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "b7edf00c4cdd4ab4a1f256aa650196ab"
format_stamp: "Formatted at 2026-09-29 18:54:46 on dist-test-slave-q2kh"
server_key: "8ae2aba14f8bb0ea52314e0f4de04f7c"
server_key_iv: "5ecad473be14603c896ed814d24f904c"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:46.204952 9139 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:54:46.217420 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:46.218101 9139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:54:46.218241 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "b7edf00c4cdd4ab4a1f256aa650196ab"
format_stamp: "Formatted at 2026-09-29 18:54:46 on dist-test-slave-q2kh"
server_key: "8ae2aba14f8bb0ea52314e0f4de04f7c"
server_key_iv: "5ecad473be14603c896ed814d24f904c"
server_key_version: "kuduclusterkey@0"
I20260929 18:54:46.218343 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:54:46.245816 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:54:46.248417 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:54:46.248636 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:54:46.248975 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:54:46.249408 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:54:46.249511 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:46.249588 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:54:46.249648 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:54:46.256808 9139 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:33369
I20260929 18:54:46.257331 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:54:46.259704 9267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:33369 every 8 connection(s)
I20260929 18:54:46.262646 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 9139
I20260929 18:54:46.262759 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:54:46.262964 29982 external_mini_cluster.cc:1428] Setting key a0c8818b65a19ac0781b642567ca6556
Sep 29 18:54:46 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708086, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:54:46.272609 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.261155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:58109 (local address 127.29.71.190:34089)
0929 18:54:46.261203 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.261205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.261542 (+ 337us) server_negotiation.cc:407] Connection header received
0929 18:54:46.262309 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.262311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.262323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.262357 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:46.263952 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.264673 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.266175 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.266512 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.266560 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.269224 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:46.269225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:46.269226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:46.269233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:46.271059 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:46.271540 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:46.271542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:46.271542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:46.271605 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:46.271886 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:46.271887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:46.271887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:46.272017 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:46.272058 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.272417 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.272506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:54:46.273180 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.260872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:34089 (local address 127.29.71.131:58109)
0929 18:54:46.261411 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:46.261457 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:46.262148 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:46.262547 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:46.262551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:46.262947 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:54:46.263853 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.263862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.264792 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.264792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.266050 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.266053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.266650 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.266651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.266784 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.267435 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:54:46.267445 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:54:46.269108 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:54:46.271158 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:46.271164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:46.271166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:46.271447 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:46.271675 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:54:46.271675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:54:46.271676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:54:46.271799 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:54:46.272100 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:54:46.272106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:54:46.272261 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 18:54:46.272818 (+ 557us) client_negotiation.cc:240] Negotiation successful
0929 18:54:46.272995 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":85,"threads_started":1}
I20260929 18:54:46.273640 9268 heartbeater.cc:347] Connected to a master server at 127.29.71.190:34089
I20260929 18:54:46.273734 9268 heartbeater.cc:464] Registering TS with master...
I20260929 18:54:46.273938 9268 heartbeater.cc:511] Master 127.29.71.190:34089 requested a full tablet report, sending...
I20260929 18:54:46.274677 8784 ts_manager.cc:194] Registered new tserver with Master: b7edf00c4cdd4ab4a1f256aa650196ab (127.29.71.131:33369)
I20260929 18:54:46.275170 8784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:58109
I20260929 18:54:46.286302 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:54:46.296455 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.288071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.29.71.190:34089)
0929 18:54:46.288127 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.288130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.288491 (+ 361us) server_negotiation.cc:407] Connection header received
0929 18:54:46.288596 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.288597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.288608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.288640 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:54:46.289446 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.290238 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.291362 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.291645 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.291692 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.293331 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:54:46.293332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:54:46.293333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:54:46.293340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:54:46.295106 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:46.295522 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:46.295523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:46.295524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:46.295583 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:54:46.295797 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:54:46.295798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:54:46.295798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:54:46.295923 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:54:46.296001 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.296259 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.296362 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 18:54:46.298794 8784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:54:46.301153 8784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:54:46.312444 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.305336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:33369 (local address 127.0.0.1:57598)
0929 18:54:46.307345 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:46.307354 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:46.307394 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:46.307700 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:46.307703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:46.307705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:46.307773 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.307775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.309379 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.309380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.311062 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.311065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.312136 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.312137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.312215 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.312229 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:54:46.312298 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:54:46.312355 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1997,"thread_start_us":69,"threads_started":1}
I20260929 18:54:46.312923 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.305879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.29.71.131:33369)
0929 18:54:46.307036 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.307042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.307452 (+ 410us) server_negotiation.cc:407] Connection header received
0929 18:54:46.307507 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.307510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.307527 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.307602 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:46.307875 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.309276 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.311183 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.311917 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.311989 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.312068 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.312788 (+ 720us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.312842 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":66,"threads_started":1}
I20260929 18:54:46.312578 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.305180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37699 (local address 127.0.0.1:55248)
0929 18:54:46.307197 (+ 2017us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:46.307208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:46.307264 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:46.308988 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:46.308989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:46.308991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:46.309046 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.309047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.310272 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.310273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.311337 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.311340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.312337 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.312338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.312418 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.312422 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:54:46.312469 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:54:46.312506 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2001,"thread_start_us":62,"threads_started":1}
I20260929 18:54:46.314198 9189 tablet_service.cc:1467] Processing CreateTablet for tablet d259a2d91e6842549b5d35e3697d77b5 (DEFAULT_TABLE table=test-table [id=46e61e9f581a4fc6b52eb56e8df07220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:54:46.314579 9189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d259a2d91e6842549b5d35e3697d77b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:46.314626 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.306036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.29.71.129:37699)
0929 18:54:46.307070 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.307075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.308763 (+ 1688us) server_negotiation.cc:407] Connection header received
0929 18:54:46.308804 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.308806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.308822 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.308916 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:46.309118 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.310145 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.311435 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.312184 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.312251 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.312303 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.314505 (+ 2202us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.314552 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":49,"threads_started":1}
I20260929 18:54:46.319033 8925 tablet_service.cc:1467] Processing CreateTablet for tablet d259a2d91e6842549b5d35e3697d77b5 (DEFAULT_TABLE table=test-table [id=46e61e9f581a4fc6b52eb56e8df07220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:54:46.319382 8925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d259a2d91e6842549b5d35e3697d77b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:46.321731 9287 tablet_bootstrap.cc:492] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Bootstrap starting.
I20260929 18:54:46.322737 9287 tablet_bootstrap.cc:654] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:46.328426 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.304946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:36019 (local address 127.0.0.1:60450)
0929 18:54:46.306509 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:46.306553 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:46.306621 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:46.324186 (+ 17565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:46.324188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:46.324191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:46.324253 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.324256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.325701 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.325703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.326908 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.326912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.328197 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.328199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.328264 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.328269 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:54:46.328300 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 18:54:46.328345 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"thread_start_us":77,"threads_started":1}
I20260929 18:54:46.329084 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.305755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.29.71.130:36019)
0929 18:54:46.324002 (+ 18247us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.324008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.324020 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:54:46.324057 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.324060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.324075 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.324343 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:46.324435 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.325596 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.327297 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.328093 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.328806 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.328925 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.328955 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.328993 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18205,"thread_start_us":86,"threads_started":1}
I20260929 18:54:46.329910 9287 log.cc:824] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:46.329851 9067 tablet_service.cc:1467] Processing CreateTablet for tablet d259a2d91e6842549b5d35e3697d77b5 (DEFAULT_TABLE table=test-table [id=46e61e9f581a4fc6b52eb56e8df07220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:54:46.330286 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d259a2d91e6842549b5d35e3697d77b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:54:46.330341 9288 tablet_bootstrap.cc:492] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549: Bootstrap starting.
I20260929 18:54:46.331010 9288 tablet_bootstrap.cc:654] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:46.331305 9288 log.cc:824] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:46.332053 9287 tablet_bootstrap.cc:492] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: No bootstrap required, opened a new log
I20260929 18:54:46.332114 9287 ts_tablet_manager.cc:1397] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260929 18:54:46.333956 9287 raft_consensus.cc:348] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.335371 9287 raft_consensus.cc:374] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:46.335465 9287 raft_consensus.cc:729] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7edf00c4cdd4ab4a1f256aa650196ab, State: Initialized, Role: FOLLOWER
I20260929 18:54:46.335659 9287 consensus_queue.cc:260] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.337211 9287 ts_tablet_manager.cc:1428] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260929 18:54:46.337378 9268 heartbeater.cc:503] Master 127.29.71.190:34089 was elected leader, sending a full tablet report...
I20260929 18:54:46.340114 9291 tablet_bootstrap.cc:492] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Bootstrap starting.
I20260929 18:54:46.341149 9291 tablet_bootstrap.cc:654] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Neither blocks nor log segments found. Creating new log.
I20260929 18:54:46.341241 9288 tablet_bootstrap.cc:492] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549: No bootstrap required, opened a new log
I20260929 18:54:46.341322 9288 ts_tablet_manager.cc:1397] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260929 18:54:46.341616 9291 log.cc:824] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:54:46.342417 9291 tablet_bootstrap.cc:492] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: No bootstrap required, opened a new log
I20260929 18:54:46.342482 9291 ts_tablet_manager.cc:1397] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:54:46.343962 9288 raft_consensus.cc:348] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.344122 9288 raft_consensus.cc:374] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:46.344151 9288 raft_consensus.cc:729] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bd5efe1124462b8ab042ab137f4549, State: Initialized, Role: FOLLOWER
I20260929 18:54:46.344262 9288 consensus_queue.cc:260] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.344502 9288 ts_tablet_manager.cc:1428] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:54:46.344411 9291 raft_consensus.cc:348] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.344558 9291 raft_consensus.cc:374] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:54:46.344587 9291 raft_consensus.cc:729] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4825eca72b05415a9ded842d29a6304a, State: Initialized, Role: FOLLOWER
I20260929 18:54:46.344609 8996 heartbeater.cc:503] Master 127.29.71.190:34089 was elected leader, sending a full tablet report...
I20260929 18:54:46.344688 9291 consensus_queue.cc:260] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.344921 9291 ts_tablet_manager.cc:1428] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:54:46.345191 9133 heartbeater.cc:503] Master 127.29.71.190:34089 was elected leader, sending a full tablet report...
Sep 29 18:54:46 dist-test-slave-q2kh krb5kdc[8308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708085, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
W20260929 18:54:46.477373 9134 tablet.cc:2369] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:54:46.487757 8997 tablet.cc:2369] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:54:46.511554 9269 tablet.cc:2369] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:54:46.578802 9294 raft_consensus.cc:482] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:54:46.578954 9294 raft_consensus.cc:504] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.579375 9294 leader_election.cc:288] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4825eca72b05415a9ded842d29a6304a (127.29.71.130:36019), b7edf00c4cdd4ab4a1f256aa650196ab (127.29.71.131:33369)
I20260929 18:54:46.583693 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.579742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:60303 (local address 127.29.71.131:33369)
0929 18:54:46.579852 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.579854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.579898 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:54:46.580040 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.580041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.580051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.580080 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:46.580361 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.581109 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.582741 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.583293 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.583341 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.583349 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.583580 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.583621 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 18:54:46.583693 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.579738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:37807 (local address 127.29.71.130:36019)
0929 18:54:46.579803 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.579805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.579924 (+ 119us) server_negotiation.cc:407] Connection header received
0929 18:54:46.580028 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.580029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.580042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.580075 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:54:46.580362 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.581100 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.582741 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.583293 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.583341 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.583349 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.583590 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.583626 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:54:46.583937 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.579540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:36019 (local address 127.29.71.129:37807)
0929 18:54:46.579868 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:46.579877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:46.579947 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:46.580149 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:46.580149 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:46.580151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:46.580271 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.580272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.581212 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.581213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.582633 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.582636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.583413 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.583414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.583492 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.583497 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:54:46.583772 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 18:54:46.583823 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":83,"thread_start_us":96,"threads_started":1}
I20260929 18:54:46.584370 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.579540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:33369 (local address 127.29.71.129:60303)
0929 18:54:46.579818 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:46.579829 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:54:46.579938 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:54:46.580149 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:54:46.580149 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:54:46.580151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:54:46.580275 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.580277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.581212 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.581213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.582633 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:54:46.582635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.583411 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.583412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:54:46.583492 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.583499 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:54:46.584255 (+ 756us) client_negotiation.cc:240] Negotiation successful
0929 18:54:46.584300 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":99,"thread_start_us":95,"threads_started":1}
I20260929 18:54:46.584409 9087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d259a2d91e6842549b5d35e3697d77b5" candidate_uuid: "91bd5efe1124462b8ab042ab137f4549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4825eca72b05415a9ded842d29a6304a" is_pre_election: true
I20260929 18:54:46.584583 9087 raft_consensus.cc:2454] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bd5efe1124462b8ab042ab137f4549 in term 0.
I20260929 18:54:46.584807 8883 leader_election.cc:302] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4825eca72b05415a9ded842d29a6304a, 91bd5efe1124462b8ab042ab137f4549; no voters:
I20260929 18:54:46.584936 9294 raft_consensus.cc:2781] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:54:46.585004 9294 raft_consensus.cc:482] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:54:46.584956 9222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d259a2d91e6842549b5d35e3697d77b5" candidate_uuid: "91bd5efe1124462b8ab042ab137f4549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" is_pre_election: true
I20260929 18:54:46.585039 9294 raft_consensus.cc:3037] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:54:46.585122 9222 raft_consensus.cc:2454] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91bd5efe1124462b8ab042ab137f4549 in term 0.
I20260929 18:54:46.585807 9294 raft_consensus.cc:504] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.585937 9294 leader_election.cc:288] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [CANDIDATE]: Term 1 election: Requested vote from peers 4825eca72b05415a9ded842d29a6304a (127.29.71.130:36019), b7edf00c4cdd4ab4a1f256aa650196ab (127.29.71.131:33369)
I20260929 18:54:46.586110 9222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d259a2d91e6842549b5d35e3697d77b5" candidate_uuid: "91bd5efe1124462b8ab042ab137f4549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab"
I20260929 18:54:46.586210 9222 raft_consensus.cc:3037] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:54:46.586220 9087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d259a2d91e6842549b5d35e3697d77b5" candidate_uuid: "91bd5efe1124462b8ab042ab137f4549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4825eca72b05415a9ded842d29a6304a"
I20260929 18:54:46.586283 9087 raft_consensus.cc:3037] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:54:46.587098 9222 raft_consensus.cc:2454] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bd5efe1124462b8ab042ab137f4549 in term 1.
I20260929 18:54:46.587165 9087 raft_consensus.cc:2454] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91bd5efe1124462b8ab042ab137f4549 in term 1.
I20260929 18:54:46.587329 8882 leader_election.cc:302] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91bd5efe1124462b8ab042ab137f4549, b7edf00c4cdd4ab4a1f256aa650196ab; no voters:
I20260929 18:54:46.587435 9294 raft_consensus.cc:2781] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:54:46.587598 9294 raft_consensus.cc:686] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [term 1 LEADER]: Becoming Leader. State: Replica: 91bd5efe1124462b8ab042ab137f4549, State: Running, Role: LEADER
I20260929 18:54:46.587697 9294 consensus_queue.cc:237] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } }
I20260929 18:54:46.588481 8784 catalog_manager.cc:5495] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91bd5efe1124462b8ab042ab137f4549 (127.29.71.129). New cstate: current_term: 1 leader_uuid: "91bd5efe1124462b8ab042ab137f4549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91bd5efe1124462b8ab042ab137f4549" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20260929 18:54:46.599692 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.596328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.29.71.129:37699)
0929 18:54:46.596375 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:54:46.596377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:54:46.596427 (+ 50us) server_negotiation.cc:407] Connection header received
0929 18:54:46.596514 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:54:46.596515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:54:46.596535 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:54:46.596643 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:54:46.596806 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.597336 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.598264 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:54:46.598497 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:54:46.598595 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:54:46.598843 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:54:46.598946 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:54:46.599054 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:54:46.599084 (+ 30us) server_negotiation.cc:299] Negotiation successful
0929 18:54:46.599107 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:54:46.601953 9087 raft_consensus.cc:1261] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 1 FOLLOWER]: Refusing update from remote peer 91bd5efe1124462b8ab042ab137f4549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:54:46.601953 9222 raft_consensus.cc:1261] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 1 FOLLOWER]: Refusing update from remote peer 91bd5efe1124462b8ab042ab137f4549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:54:46.602391 9298 consensus_queue.cc:1035] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7edf00c4cdd4ab4a1f256aa650196ab" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:54:46.602697 9298 consensus_queue.cc:1035] T d259a2d91e6842549b5d35e3697d77b5 P 91bd5efe1124462b8ab042ab137f4549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4825eca72b05415a9ded842d29a6304a" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:54:46.608596 9301 mvcc.cc:205] Tried to move back new op lower bound from 7334740322718371840 to 7334740322663710720. Current Snapshot: MvccSnapshot[applied={T|T < 7334740322718371840}]
I20260929 18:54:46.609006 9302 mvcc.cc:205] Tried to move back new op lower bound from 7334740322718371840 to 7334740322663710720. Current Snapshot: MvccSnapshot[applied={T|T < 7334740322718371840}]
I20260929 18:54:46.609184 9304 mvcc.cc:205] Tried to move back new op lower bound from 7334740322718371840 to 7334740322663710720. Current Snapshot: MvccSnapshot[applied={T|T < 7334740322718371840}]
I20260929 18:54:46.612252 8783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:54:46.612366 8783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:54:46.613078 8783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc060fd7713c4d47ad3087d63a5a5119: Sending DeleteTablet for 3 replicas of tablet d259a2d91e6842549b5d35e3697d77b5
I20260929 18:54:46.613515 8925 tablet_service.cc:1514] Processing DeleteTablet for tablet d259a2d91e6842549b5d35e3697d77b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:54:46 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:55248
I20260929 18:54:46.613686 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 8865
I20260929 18:54:46.613837 9067 tablet_service.cc:1514] Processing DeleteTablet for tablet d259a2d91e6842549b5d35e3697d77b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:54:46 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:60450
I20260929 18:54:46.615046 9189 tablet_service.cc:1514] Processing DeleteTablet for tablet d259a2d91e6842549b5d35e3697d77b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:54:46 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:57598
I20260929 18:54:46.615774 9311 tablet_replica.cc:330] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: stopping tablet replica
I20260929 18:54:46.615909 9311 raft_consensus.cc:2229] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:54:46.616030 9311 raft_consensus.cc:2258] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:54:46.616209 9312 tablet_replica.cc:330] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: stopping tablet replica
I20260929 18:54:46.616365 9312 raft_consensus.cc:2229] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:54:46.616518 9312 raft_consensus.cc:2258] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:54:46.616530 9311 ts_tablet_manager.cc:1905] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:54:46.618654 9311 ts_tablet_manager.cc:1918] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:54:46.618783 9311 log.cc:1196] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/d259a2d91e6842549b5d35e3697d77b5
I20260929 18:54:46.619132 9311 ts_tablet_manager.cc:1939] T d259a2d91e6842549b5d35e3697d77b5 P 4825eca72b05415a9ded842d29a6304a: Deleting consensus metadata
I20260929 18:54:46.619618 8769 catalog_manager.cc:4841] TS 4825eca72b05415a9ded842d29a6304a (127.29.71.130:36019): tablet d259a2d91e6842549b5d35e3697d77b5 (table test-table [id=46e61e9f581a4fc6b52eb56e8df07220]) successfully deleted
I20260929 18:54:46.620664 9312 ts_tablet_manager.cc:1905] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:54:46.622071 9312 ts_tablet_manager.cc:1918] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:54:46.622223 9312 log.cc:1196] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/d259a2d91e6842549b5d35e3697d77b5
W20260929 18:54:46.622210 8768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:37699 (error 108)
I20260929 18:54:46.622411 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 9002
I20260929 18:54:46.623041 9312 ts_tablet_manager.cc:1939] T d259a2d91e6842549b5d35e3697d77b5 P b7edf00c4cdd4ab4a1f256aa650196ab: Deleting consensus metadata
I20260929 18:54:46.623421 8768 catalog_manager.cc:4841] TS b7edf00c4cdd4ab4a1f256aa650196ab (127.29.71.131:33369): tablet d259a2d91e6842549b5d35e3697d77b5 (table test-table [id=46e61e9f581a4fc6b52eb56e8df07220]) successfully deleted
I20260929 18:54:46.627506 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 9139
I20260929 18:54:46.628505 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:54:46.628370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37699 (local address 127.0.0.1:55266)
0929 18:54:46.628424 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 18:54:46.628464 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 18:54:46.628700 8768 catalog_manager.cc:4584] TS 91bd5efe1124462b8ab042ab137f4549 (127.29.71.129:37699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d259a2d91e6842549b5d35e3697d77b5: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37699: connect: Connection refused (error 111)
I20260929 18:54:46.632006 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 8752
I20260929 18:54:46.637444 29982 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:54:46.982086 29982 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:54:46.982240 29982 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:54:47.326571 29982 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:54:47.326 UTC [8338] LOG: received smart shutdown request
2026-09-29 18:54:47.330 UTC [8338] LOG: background worker "logical replication launcher" (PID 8347) exited with exit code 1
2026-09-29 18:54:47.330 UTC [8342] LOG: shutting down
2026-09-29 18:54:47.504 UTC [8338] LOG: database system is shut down
2026-09-29T18:54:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46637 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:54:47 dist-test-slave-q2kh krb5kdc[9322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:54:47 dist-test-slave-q2kh krb5kdc[9322](info): set up 2 sockets
Sep 29 18:54:47 dist-test-slave-q2kh krb5kdc[9322](info): commencing operation
krb5kdc: starting...
W20260929 18:54:48.682130 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:54:48 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:54:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/rangeradmin_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/rangeradmin_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/rangerlookup_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/rangerlookup_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
I20260929 18:54:48.853645 29982 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWZtMeW/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/postgres -l logfile start
2026-09-29 18:54:51.427 UTC [9352] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:54:51.427 UTC [9352] LOG: listening on IPv4 address "127.29.71.148", port 51323
2026-09-29 18:54:51.430 UTC [9352] LOG: listening on Unix socket "/tmp/.s.PGSQL.51323"
2026-09-29 18:54:51.451 UTC [9355] LOG: database system was shut down at 2026-09-29 18:54:49 UTC
2026-09-29 18:54:51.455 UTC [9352] LOG: database system is ready to accept connections
I20260929 18:54:52.398505 29982 mini_postgres.cc:96] Postgres bound to 51323
2026-09-29 18:54:52.404 UTC [9363] FATAL: database "slave" does not exist
127.29.71.148:51323 - accepting connections
I20260929 18:54:52.405161 29982 mini_ranger.cc:161] Starting Ranger...
I20260929 18:54:52.420369 29982 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:54:52.939391 29982 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:54:52.939476 29982 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-admin
2026-09-29 18:54:53,224 [I] DB FLAVOR :POSTGRES
2026-09-29 18:54:53,224 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:54:53,224 [I] Checking connection..
2026-09-29 18:54:53,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:54:53,485 [I] Checking connection passed.
2026-09-29 18:54:53,485 [I] --------- Verifying version history table ---------
2026-09-29 18:54:53,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:54:53,726 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:54:53,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:54:53,970 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:54:53,970 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:54:53,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:54:54.235 UTC [9427] WARNING: there is no transaction in progress
2026-09-29 18:54:54,241 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:54:54,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:54:54,473 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:54:54,474 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:54:54,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:54:54,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:54:54,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q2kh.c.gcp-upstream.internal','N') ;"
2026-09-29 18:54:55,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:54:55,472 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:54:55,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:54:55,726 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:54:55,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:54:55,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:54:56,225 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:54:56,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:54:57.201 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.217 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.235 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.251 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.269 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.290 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.356 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.362 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.377 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.388 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.398 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.411 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.421 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.428 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.436 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:57.443 UTC [9535] WARNING: there is no transaction in progress
2026-09-29 18:54:58,109 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:54:58,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:54:58,331 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:54:58,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:54:58,558 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:54:58,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:54:58,792 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:54:58,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:54:59,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:54:59,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q2kh.c.gcp-upstream.internal';"
2026-09-29 18:54:59,536 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:54:59,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:54:59,798 [I] DB_PATCHES have already been applied
I20260929 18:54:59.803982 29982 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWZtMeW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:54:59.804050 29982 mini_ranger.cc:193] Using host: 127.29.71.148
I20260929 18:54:59.807356 29982 mini_ranger.cc:227] Ranger admin URL: http://127.29.71.148:56889
Sep 29, 2026 6:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:55:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:55:00.900681 29982 mini_ranger.cc:160] Time spent starting Ranger: real 8.496s user 0.000s sys 0.009s
Sep 29, 2026 6:55:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:55:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 6:55:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:55:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:55:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:55:21.940076 29982 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.71.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.71.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/rangerkms_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.71.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/rangerkms_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.71.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.71.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.71.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/HTTP_127.29.71.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:55:22 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708122, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:55:22.021023 29982 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:55:22.030498 29982 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:55:22.936003 29982 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:55:22.936090 29982 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms
I20260929 18:55:22.936110 29982 mini_ranger_kms.cc:227] Using postgres at 127.29.71.148:51323
2026-09-29 18:55:23,135 [I] DB FLAVOR :POSTGRES
2026-09-29 18:55:23,136 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:55:23,136 [I] Checking connection
2026-09-29 18:55:23,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:55:23,377 [I] connection success
2026-09-29 18:55:23,377 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:55:23,377 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:55:23,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:55:23,609 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:55:23,609 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:55:23,609 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:55:23,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.71.148:51323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:55:23,858 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:55:24.331295 29982 mini_ranger_kms.cc:310] Created kms service
I20260929 18:55:24.548470 29982 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:55:24.594403 29982 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:55:24 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708122, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:24.965593 29982 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:55:24.965786 29982 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWZtMeW/build/release/bin/postgresql.jar:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWZtMeW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:55:24.965823 29982 mini_ranger_kms.cc:240] Using host: 127.29.71.148
I20260929 18:55:24.969482 29982 mini_ranger_kms.cc:276] Ranger KMS PID: 9715
I20260929 18:55:24.969574 29982 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.71.148:36687
Sep 29, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:55:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:55:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:55:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:55:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:55:27.197985 29982 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.177s user 0.007s sys 0.174s
I20260929 18:55:27.198081 29982 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:55:27.198124 29982 mini_ranger_kms.cc:401] 127.29.71.148:36687/kms/v1/keys
Sep 29 18:55:27 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708127, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.71.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:55:27 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708127, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.71.148@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
Sep 29 18:55:28 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708127, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.71.148@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
Sep 29 18:55:30 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:55:30.410132 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:44247
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:44247
--ranger_config_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:30.492682 9766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:30.492885 9766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:30.492906 9766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:30.494096 9766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:30.494190 9766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:30.494222 9766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:30.494264 9766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:30.494282 9766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:30.495823 9766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.71.148:36687/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:44247
--ranger_config_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:44247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:30.496060 9766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:30.496305 9766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:30.501559 9771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:30.501590 9772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:30.501667 9766 server_base.cc:1034] running on GCE node
W20260929 18:55:30.501816 9774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:30.502069 9766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:30.502435 9766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:30.503615 9766 hybrid_clock.cc:648] HybridClock initialized: now 1790708130503602 us; error 28 us; skew 500 ppm
Sep 29 18:55:30 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708130, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:30.513943 9766 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:30.514552 9766 webserver.cc:460] Webserver started at http://127.29.71.190:41375/ using document root <none> and password file <none>
I20260929 18:55:30.514794 9766 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:30.514832 9766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:30.514968 9766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:55:30 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708130, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:30.560734 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "db40a02cb3da49b99312cfec6540faa5"
format_stamp: "Formatted at 2026-09-29 18:55:30 on dist-test-slave-q2kh"
server_key: "4526428cfdc2b46ddcd80682cd67a0e2"
server_key_iv: "4c540d707ac4b6d75aa4727693710649"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:30.561131 9766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "db40a02cb3da49b99312cfec6540faa5"
format_stamp: "Formatted at 2026-09-29 18:55:30 on dist-test-slave-q2kh"
server_key: "4526428cfdc2b46ddcd80682cd67a0e2"
server_key_iv: "4c540d707ac4b6d75aa4727693710649"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:30.562638 9766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:55:30.575289 9781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:30.575508 9766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260929 18:55:30.575618 9766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "db40a02cb3da49b99312cfec6540faa5"
format_stamp: "Formatted at 2026-09-29 18:55:30 on dist-test-slave-q2kh"
server_key: "4526428cfdc2b46ddcd80682cd67a0e2"
server_key_iv: "4c540d707ac4b6d75aa4727693710649"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:30.575726 9766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:30.609520 9766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:30.611600 9766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:30.611778 9766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:30.616708 9766 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:44247
I20260929 18:55:30.616784 9833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:44247 every 8 connection(s)
I20260929 18:55:30.617151 9766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:30.617930 9834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:30.620879 9834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Bootstrap starting.
I20260929 18:55:30.621536 9834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:30.621829 9834 log.cc:824] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:30.622628 9834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: No bootstrap required, opened a new log
I20260929 18:55:30.625313 9834 raft_consensus.cc:348] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:30.625499 9834 raft_consensus.cc:374] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:30.625532 9834 raft_consensus.cc:729] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db40a02cb3da49b99312cfec6540faa5, State: Initialized, Role: FOLLOWER
I20260929 18:55:30.625667 9834 consensus_queue.cc:260] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:30.625734 9834 raft_consensus.cc:388] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:55:30.625761 9834 raft_consensus.cc:482] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:55:30.625804 9834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:30.626420 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 9766
I20260929 18:55:30.626539 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:55:30.626542 9834 raft_consensus.cc:504] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:30.626677 9834 leader_election.cc:302] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db40a02cb3da49b99312cfec6540faa5; no voters:
I20260929 18:55:30.626737 29982 external_mini_cluster.cc:1428] Setting key 6f0c68a6d7e89e47f6f22ca8e74d8ac8
I20260929 18:55:30.626844 9834 leader_election.cc:288] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:55:30.626888 9837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:30.627134 9837 raft_consensus.cc:686] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 LEADER]: Becoming Leader. State: Replica: db40a02cb3da49b99312cfec6540faa5, State: Running, Role: LEADER
I20260929 18:55:30.627194 9834 sys_catalog.cc:564] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:30.627305 9837 consensus_queue.cc:237] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:30.627630 9837 sys_catalog.cc:455] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db40a02cb3da49b99312cfec6540faa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } } }
I20260929 18:55:30.627683 9838 sys_catalog.cc:455] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db40a02cb3da49b99312cfec6540faa5. Latest consensus state: current_term: 1 leader_uuid: "db40a02cb3da49b99312cfec6540faa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } } }
I20260929 18:55:30.627722 9837 sys_catalog.cc:458] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:30.627945 9838 sys_catalog.cc:458] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:30.628801 9844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:30.628950 9834 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:55:30.629403 9844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:55:30 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:30.642752 9844 catalog_manager.cc:1347] Generated new cluster ID: 5b265ed4dfb74673ae2d2aef70fec6a6
I20260929 18:55:30.642907 9844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:55:30.646386 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:30.627435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.29.71.190:44247)
0929 18:55:30.627610 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:30.627619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:30.627970 (+ 351us) server_negotiation.cc:407] Connection header received
0929 18:55:30.628213 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:30.628220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:30.628280 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:30.628394 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:30.629407 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:30.633619 (+ 4212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:30.634825 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:30.635713 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:30.635839 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:30.638870 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:30.638875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:30.638876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:30.638888 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:30.640953 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:30.641354 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:30.641357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:30.641358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:30.641471 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:30.641663 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:30.641664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:30.641664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:30.641987 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:30.642127 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:30.642669 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 18:55:30.642807 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260929 18:55:30.656416 9844 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:55:30.657235 9844 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:55:30.674351 9844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Generated new TSK 0
I20260929 18:55:30.674666 9844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:55:31.455008 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44247
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:31.569559 9879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:31.569860 9879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:31.569903 9879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:31.571703 9879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:31.571769 9879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:31.571811 9879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:55:31.574090 9879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.71.148:36687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:31.574411 9879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:31.574719 9879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:31.581076 9887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:31.581104 9884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:31.581185 9885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:31.581516 9879 server_base.cc:1034] running on GCE node
I20260929 18:55:31.581732 9879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:31.582041 9879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:31.583210 9879 hybrid_clock.cc:648] HybridClock initialized: now 1790708131583187 us; error 35 us; skew 500 ppm
Sep 29 18:55:31 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708131, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:31.586692 9879 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:55:31.587265 9879 webserver.cc:460] Webserver started at http://127.29.71.129:34179/ using document root <none> and password file <none>
I20260929 18:55:31.587595 9879 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:31.587710 9879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:31.587906 9879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:55:31 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708131, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:31.602959 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "1c73f0bd4e3f47598c067297f8b8b33c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "52c3620d2f4e8e9d85e82f9d5c47150f"
server_key_iv: "f62dce692655abd5704b24a6d7e4996f"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:31.603434 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "1c73f0bd4e3f47598c067297f8b8b33c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "52c3620d2f4e8e9d85e82f9d5c47150f"
server_key_iv: "f62dce692655abd5704b24a6d7e4996f"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:31.605325 9879 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 18:55:31.619696 9894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:31.620018 9879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:55:31.620131 9879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "1c73f0bd4e3f47598c067297f8b8b33c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "52c3620d2f4e8e9d85e82f9d5c47150f"
server_key_iv: "f62dce692655abd5704b24a6d7e4996f"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:31.620224 9879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:31.641248 9879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:31.643810 9879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:31.643998 9879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:31.644264 9879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:31.644680 9879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:31.644735 9879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:31.644791 9879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:31.644840 9879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:31.653309 9879 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:44333
I20260929 18:55:31.653760 9879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:55:31.659636 10009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:44333 every 8 connection(s)
I20260929 18:55:31.663117 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 9879
I20260929 18:55:31.663213 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:55:31.663421 29982 external_mini_cluster.cc:1428] Setting key 78e948270564a4b7afc205b7766d3f25
Sep 29 18:55:31 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708131, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:31.678997 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:31.660342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44247 (local address 127.29.71.129:59007)
0929 18:55:31.660540 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:31.660592 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:31.661130 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:31.661428 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:31.661431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:31.661625 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:31.661826 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:31.661830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.662609 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.662610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:31.666093 (+ 3483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:31.666097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.670298 (+ 4201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.670299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:31.670471 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:31.671131 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:31.671142 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:31.672895 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:31.677221 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:31.677228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:31.677231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:31.677542 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:31.677760 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:31.677760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:31.677761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:31.677921 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:31.678254 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:31.678261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:31.678438 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 18:55:31.678541 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:55:31.678744 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260929 18:55:31.679827 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:31.660629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:59007 (local address 127.29.71.190:44247)
0929 18:55:31.660767 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:31.660769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:31.660780 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:55:31.661251 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:31.661253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:31.661268 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:31.661347 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:31.661919 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.662513 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.667211 (+ 4698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.667548 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.667617 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:31.675174 (+ 7557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:31.675175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:31.675176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:31.675184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:31.677114 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:31.677631 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:31.677632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:31.677632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:31.677692 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:31.678011 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:31.678012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:31.678012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:31.678175 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:31.679593 (+ 1418us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:31.679638 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 18:55:31.679731 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260929 18:55:31.682515 10010 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44247
I20260929 18:55:31.682615 10010 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:31.682802 10010 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:31.683423 9798 ts_manager.cc:194] Registered new tserver with Master: 1c73f0bd4e3f47598c067297f8b8b33c (127.29.71.129:44333)
I20260929 18:55:31.684162 9798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:59007
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:55:31.709862 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44247
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:55:31 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708131, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:55:31.842711 10016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:31.843045 10016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:31.843128 10016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:31.844889 10016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:31.844998 10016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:31.845080 10016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:55:31.847391 10016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.71.148:36687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:31.847853 10016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:31.848174 10016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:31.855187 10026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:31.855424 10024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:31.855163 10023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:31.856604 10016 server_base.cc:1034] running on GCE node
I20260929 18:55:31.856817 10016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:31.857061 10016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:31.864384 10016 hybrid_clock.cc:648] HybridClock initialized: now 1790708131864347 us; error 37 us; skew 500 ppm
Sep 29 18:55:31 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708131, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:31.867786 10016 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:55:31.868292 10016 webserver.cc:460] Webserver started at http://127.29.71.130:46727/ using document root <none> and password file <none>
I20260929 18:55:31.868569 10016 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:31.868619 10016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:31.868741 10016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:55:31 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708131, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:31.885946 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "b109470f33b74ee0920218bb1ac9f21c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "2c2bf9402d4551f4b0b0577f3fcb7c82"
server_key_iv: "ec7b5428d12f77ed3635f7149210565a"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:31.886461 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "b109470f33b74ee0920218bb1ac9f21c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "2c2bf9402d4551f4b0b0577f3fcb7c82"
server_key_iv: "ec7b5428d12f77ed3635f7149210565a"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:31.888448 10016 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:55:31.905300 10033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:31.905566 10016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:31.905666 10016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "b109470f33b74ee0920218bb1ac9f21c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "2c2bf9402d4551f4b0b0577f3fcb7c82"
server_key_iv: "ec7b5428d12f77ed3635f7149210565a"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:31.905758 10016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:31.931229 10016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:31.933862 10016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:31.934074 10016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:31.934407 10016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:31.934793 10016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:31.934883 10016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:31.934954 10016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:31.935014 10016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:31.941632 10016 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:39385
I20260929 18:55:31.942471 10016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:55:31.945333 10147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:39385 every 8 connection(s)
I20260929 18:55:31.950385 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10016
I20260929 18:55:31.950497 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:55:31.950711 29982 external_mini_cluster.cc:1428] Setting key 0601d36a076f7bde9a9a7d5515e156a8
Sep 29 18:55:31 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708131, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:31.959755 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:31.946259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:50093 (local address 127.29.71.190:44247)
0929 18:55:31.946306 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:31.946308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:31.946809 (+ 501us) server_negotiation.cc:407] Connection header received
0929 18:55:31.947541 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:31.947542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:31.947560 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:31.947591 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:31.948478 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.949254 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.953079 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.953406 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.953455 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:31.956266 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:31.956268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:31.956268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:31.956276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:31.958030 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:31.958581 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:31.958582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:31.958582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:31.958644 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:31.958960 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:31.958961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:31.958961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:31.959106 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:31.959165 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:31.959561 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 18:55:31.959654 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:55:31.960587 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:31.945973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44247 (local address 127.29.71.130:50093)
0929 18:55:31.946675 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:31.946719 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:31.947407 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:31.947790 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:31.947794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:31.948068 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:31.948362 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:31.948369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.949372 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.949373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:31.950869 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:31.952980 (+ 2111us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:31.953581 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:31.953582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:31.953731 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:31.954411 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:31.954421 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:31.956153 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:31.958130 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:31.958136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:31.958138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:31.958486 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:31.958714 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:31.958715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:31.958715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:31.958872 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:31.959202 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:31.959208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:31.959386 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 18:55:31.960157 (+ 771us) client_negotiation.cc:240] Negotiation successful
0929 18:55:31.960336 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":88,"threads_started":1}
I20260929 18:55:31.961310 10148 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44247
I20260929 18:55:31.961409 10148 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:31.961611 10148 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:31.962059 9798 ts_manager.cc:194] Registered new tserver with Master: b109470f33b74ee0920218bb1ac9f21c (127.29.71.130:39385)
I20260929 18:55:31.962616 9798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:50093
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:55:31.994670 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44247
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:32.108755 10153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:32.109143 10153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:32.109238 10153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:32.111161 10153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:32.111321 10153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:32.111423 10153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:55:32.113710 10153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.71.148:36687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:32.114244 10153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:32.114627 10153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:32.130021 10159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.130380 10161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.138396 10158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:32.140403 10153 server_base.cc:1034] running on GCE node
I20260929 18:55:32.140839 10153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:32.141230 10153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:32.143160 10153 hybrid_clock.cc:648] HybridClock initialized: now 1790708132143131 us; error 42 us; skew 500 ppm
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:32.148697 10153 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:55:32.149361 10153 webserver.cc:460] Webserver started at http://127.29.71.131:46617/ using document root <none> and password file <none>
I20260929 18:55:32.149685 10153 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:32.149755 10153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:32.149900 10153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:32.165702 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "7bbdfee146de493c9d28786564cb0682"
format_stamp: "Formatted at 2026-09-29 18:55:32 on dist-test-slave-q2kh"
server_key: "30345306c3c3a62ebb6040423b19fef2"
server_key_iv: "58d41ef0b9c27e379c37b4df0b36f4ec"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:32.166220 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "7bbdfee146de493c9d28786564cb0682"
format_stamp: "Formatted at 2026-09-29 18:55:32 on dist-test-slave-q2kh"
server_key: "30345306c3c3a62ebb6040423b19fef2"
server_key_iv: "58d41ef0b9c27e379c37b4df0b36f4ec"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:32.168161 10153 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260929 18:55:32.202298 10168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.205243 10153 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260929 18:55:32.205379 10153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "7bbdfee146de493c9d28786564cb0682"
format_stamp: "Formatted at 2026-09-29 18:55:32 on dist-test-slave-q2kh"
server_key: "30345306c3c3a62ebb6040423b19fef2"
server_key_iv: "58d41ef0b9c27e379c37b4df0b36f4ec"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:32.205518 10153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:32.218082 10153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:32.220530 10153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:32.220671 10153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:32.220889 10153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:32.221199 10153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:32.221231 10153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.221259 10153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:32.221276 10153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.227735 10153 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:34029
I20260929 18:55:32.228179 10153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:55:32.235617 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10153
I20260929 18:55:32.235742 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:55:32.235930 29982 external_mini_cluster.cc:1428] Setting key 1a1e792ce9e98c04914a6a681133d4d8
I20260929 18:55:32.239009 10281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:34029 every 8 connection(s)
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:32.255210 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:32.240066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:36771 (local address 127.29.71.190:44247)
0929 18:55:32.240108 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:32.240110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:32.241480 (+ 1370us) server_negotiation.cc:407] Connection header received
0929 18:55:32.242961 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:32.242962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:32.242977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:32.243008 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:32.244144 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.244926 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.246462 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.246751 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.246798 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:32.251433 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:32.251435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:32.251435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:32.251444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:32.253643 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:32.254126 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:32.254127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:32.254128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:32.254213 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:32.254492 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:32.254493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:32.254494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:32.254620 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:32.254669 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:32.255017 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 18:55:32.255117 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:55:32.255756 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:32.239776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44247 (local address 127.29.71.131:36771)
0929 18:55:32.241202 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:32.241255 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:32.242828 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:32.243213 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:32.243218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:32.243514 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:32.243865 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:32.244031 (+ 166us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.245044 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.245045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:32.246362 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:32.246365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.246909 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.246910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:32.247049 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:32.247705 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:32.247715 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:32.251300 (+ 3585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:32.253743 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:32.253750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:32.253753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:32.254040 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:32.254281 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:32.254281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:32.254282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:32.254411 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:32.254701 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:32.254707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:32.254867 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 18:55:32.255349 (+ 482us) client_negotiation.cc:240] Negotiation successful
0929 18:55:32.255525 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"thread_start_us":99,"threads_started":1}
I20260929 18:55:32.256179 10282 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44247
I20260929 18:55:32.256285 10282 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:32.256485 10282 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:32.256830 9798 ts_manager.cc:194] Registered new tserver with Master: 7bbdfee146de493c9d28786564cb0682 (127.29.71.131:34029)
I20260929 18:55:32.257290 9798 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:36771
I20260929 18:55:32.263068 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:55:32.263746 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 9879
I20260929 18:55:32.272169 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10016
I20260929 18:55:32.281631 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10153
I20260929 18:55:32.291875 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 9766
I20260929 18:55:32.296603 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:44247
--webserver_interface=127.29.71.190
--webserver_port=41375
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:44247
--ranger_config_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:32.378600 10285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:32.378835 10285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:32.378873 10285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:32.380569 10285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:32.380645 10285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:32.380671 10285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:32.380700 10285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:32.380726 10285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:32.382432 10285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:44247
--ranger_config_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:44247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=41375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:32.382694 10285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:32.382915 10285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:32.388978 10293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.388964 10291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:32.389216 10285 server_base.cc:1034] running on GCE node
W20260929 18:55:32.388950 10290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:32.389541 10285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:32.389869 10285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:32.391032 10285 hybrid_clock.cc:648] HybridClock initialized: now 1790708132391020 us; error 27 us; skew 500 ppm
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:32.393596 10285 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:32.394091 10285 webserver.cc:460] Webserver started at http://127.29.71.190:41375/ using document root <none> and password file <none>
I20260929 18:55:32.394369 10285 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:32.394423 10285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:32.453276 10285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:32.454032 10301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.454243 10285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:32.454358 10285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "db40a02cb3da49b99312cfec6540faa5"
format_stamp: "Formatted at 2026-09-29 18:55:30 on dist-test-slave-q2kh"
server_key: "4526428cfdc2b46ddcd80682cd67a0e2"
server_key_iv: "4c540d707ac4b6d75aa4727693710649"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:32.454821 10285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:32.475263 10285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:32.476867 10285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:32.476981 10285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:32.481225 10285 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:44247
I20260929 18:55:32.481267 10353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:44247 every 8 connection(s)
I20260929 18:55:32.481642 10285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:32.482724 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10285
I20260929 18:55:32.483008 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:44333
--local_ip_for_outbound_sockets=127.29.71.129
--tserver_master_addrs=127.29.71.190:44247
--webserver_port=34179
--webserver_interface=127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:32.483026 10354 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:55:32.483714 10354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Bootstrap starting.
I20260929 18:55:32.491777 10354 log.cc:824] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:32.494076 10354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:55:32.494441 10354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Bootstrap complete.
I20260929 18:55:32.498569 10354 raft_consensus.cc:348] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:32.498976 10354 raft_consensus.cc:729] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db40a02cb3da49b99312cfec6540faa5, State: Initialized, Role: FOLLOWER
I20260929 18:55:32.499156 10354 consensus_queue.cc:260] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:32.499243 10354 raft_consensus.cc:388] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:55:32.499289 10354 raft_consensus.cc:482] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:55:32.499344 10354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:55:32.500628 10354 raft_consensus.cc:504] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:32.500734 10354 leader_election.cc:302] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db40a02cb3da49b99312cfec6540faa5; no voters:
I20260929 18:55:32.500905 10354 leader_election.cc:288] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:55:32.500980 10358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:55:32.501137 10354 sys_catalog.cc:564] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:32.501180 10358 raft_consensus.cc:686] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [term 2 LEADER]: Becoming Leader. State: Replica: db40a02cb3da49b99312cfec6540faa5, State: Running, Role: LEADER
I20260929 18:55:32.501281 10358 consensus_queue.cc:237] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } }
I20260929 18:55:32.501637 10359 sys_catalog.cc:455] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db40a02cb3da49b99312cfec6540faa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } } }
I20260929 18:55:32.501714 10359 sys_catalog.cc:458] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:32.501915 10360 sys_catalog.cc:455] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db40a02cb3da49b99312cfec6540faa5. Latest consensus state: current_term: 2 leader_uuid: "db40a02cb3da49b99312cfec6540faa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db40a02cb3da49b99312cfec6540faa5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44247 } } }
I20260929 18:55:32.502072 10360 sys_catalog.cc:458] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:32.502297 10367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:32.502655 10354 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:55:32.502885 10367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:55:32.505133 10367 catalog_manager.cc:1259] Loaded cluster ID: 5b265ed4dfb74673ae2d2aef70fec6a6
I20260929 18:55:32.505285 10367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:55:32.507423 10367 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:55:32.507634 10367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Loaded TSK: 0
I20260929 18:55:32.507874 10367 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:55:32.606233 10356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:32.606516 10356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:32.606551 10356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:32.608263 10356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:32.608322 10356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:32.608372 10356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:55:32.610565 10356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:44333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=34179
--tserver_master_addrs=127.29.71.190:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:32.610790 10356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:32.611047 10356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:32.617192 10394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:32.617508 10356 server_base.cc:1034] running on GCE node
W20260929 18:55:32.617615 10391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.617700 10392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:32.617908 10356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:32.618194 10356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:32.619390 10356 hybrid_clock.cc:648] HybridClock initialized: now 1790708132619359 us; error 45 us; skew 500 ppm
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:32.623143 10356 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:55:32.623662 10356 webserver.cc:460] Webserver started at http://127.29.71.129:34179/ using document root <none> and password file <none>
I20260929 18:55:32.623917 10356 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:32.623996 10356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:32.646281 10356 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260929 18:55:32.648262 10402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.648532 10356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260929 18:55:32.648648 10356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "1c73f0bd4e3f47598c067297f8b8b33c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "52c3620d2f4e8e9d85e82f9d5c47150f"
server_key_iv: "f62dce692655abd5704b24a6d7e4996f"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:32.649151 10356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:32.661791 10356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:32.664045 10356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:32.664162 10356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:32.664435 10356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:32.664821 10356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:32.664876 10356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.664918 10356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:32.664944 10356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.672214 10356 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:44333
I20260929 18:55:32.672258 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:44333 every 8 connection(s)
I20260929 18:55:32.672639 10356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:55:32.674511 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10356
I20260929 18:55:32.674814 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:39385
--local_ip_for_outbound_sockets=127.29.71.130
--tserver_master_addrs=127.29.71.190:44247
--webserver_port=46727
--webserver_interface=127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:32.689441 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:32.673348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44247 (local address 127.29.71.129:57877)
0929 18:55:32.673622 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:32.673668 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:32.674210 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:32.674785 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:32.674788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:32.675044 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:32.675307 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:32.675312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.676723 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.676725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:32.677793 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:32.677796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.678556 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.678557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:32.678686 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:32.680604 (+ 1918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:32.680613 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:32.682509 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:32.687521 (+ 5012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:32.687531 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:32.687533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:32.687861 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:32.688165 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:32.688165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:32.688166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:32.688292 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:32.688768 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:32.688774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:32.688944 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 18:55:32.689032 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:55:32.689195 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
I20260929 18:55:32.690512 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:32.673512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:57877 (local address 127.29.71.190:44247)
0929 18:55:32.673715 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:32.673723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:32.673775 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:55:32.674461 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:32.674465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:32.674510 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:32.674647 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:32.675421 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.676594 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.677895 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.678397 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.678573 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:32.685288 (+ 6715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:32.685293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:32.685293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:32.685306 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:32.687406 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:32.687968 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:32.687972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:32.687972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:32.688087 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:32.688368 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:32.688369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:32.688370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:32.688678 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:32.690068 (+ 1390us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:32.690146 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:55:32.690306 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260929 18:55:32.691434 10516 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44247
I20260929 18:55:32.691527 10516 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:32.691710 10516 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
W20260929 18:55:32.692612 10516 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:44247 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:55:32.692705 10516 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:32.692767 10516 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:32.693164 10516 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:32.693236 10516 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
W20260929 18:55:32.693570 10516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:55:32.789090 10520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:32.789391 10520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:32.789431 10520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:32.791170 10520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:32.791240 10520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:32.791296 10520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:55:32.792928 10520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:39385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=46727
--tserver_master_addrs=127.29.71.190:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:32.793198 10520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:32.793448 10520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:32.798707 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.798702 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.799088 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:32.799260 10520 server_base.cc:1034] running on GCE node
I20260929 18:55:32.799554 10520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:32.799808 10520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:32.803069 10520 hybrid_clock.cc:648] HybridClock initialized: now 1790708132803036 us; error 49 us; skew 500 ppm
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:32.805897 10520 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:55:32.806517 10520 webserver.cc:460] Webserver started at http://127.29.71.130:46727/ using document root <none> and password file <none>
I20260929 18:55:32.806787 10520 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:32.806857 10520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:32.832882 10520 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:55:32.834043 10536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.834263 10520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:32.834369 10520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "b109470f33b74ee0920218bb1ac9f21c"
format_stamp: "Formatted at 2026-09-29 18:55:31 on dist-test-slave-q2kh"
server_key: "2c2bf9402d4551f4b0b0577f3fcb7c82"
server_key_iv: "ec7b5428d12f77ed3635f7149210565a"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:32.834870 10520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:32.849633 10520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:32.852521 10520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:32.852697 10520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:32.852990 10520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:32.853403 10520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:32.853457 10520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.853487 10520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:32.853504 10520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:32.861128 10520 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:39385
I20260929 18:55:32.861617 10520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:55:32.862182 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:39385 every 8 connection(s)
I20260929 18:55:32.862366 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10520
I20260929 18:55:32.862661 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.71.148:36687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:34029
--local_ip_for_outbound_sockets=127.29.71.131
--tserver_master_addrs=127.29.71.190:44247
--webserver_port=46617
--webserver_interface=127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:32.873445 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:32.862467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:51965 (local address 127.29.71.190:44247)
0929 18:55:32.862513 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:32.862515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:32.862858 (+ 343us) server_negotiation.cc:407] Connection header received
0929 18:55:32.863684 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:32.863685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:32.863701 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:32.863735 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:32.864633 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.865436 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.867031 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.867312 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.867411 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:32.869881 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:32.869882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:32.869883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:32.869892 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:32.871716 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:32.872256 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:32.872257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:32.872258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:32.872325 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:32.872654 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:32.872655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:32.872655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:32.872804 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:32.872862 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:32.873252 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 18:55:32.873348 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:55:32.874270 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:32.862492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44247 (local address 127.29.71.130:51965)
0929 18:55:32.862696 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:32.862747 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:32.863525 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:32.863951 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:32.863955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:32.864214 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:32.864507 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:32.864515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.865561 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.865562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:32.866923 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:32.866926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:32.867438 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:32.867439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:32.867560 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:32.868171 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:32.868180 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:32.869771 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:32.871816 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:32.871837 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:32.871839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:32.872138 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:32.872411 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:32.872412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:32.872412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:32.872563 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:32.872916 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:32.872923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:32.873086 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 18:55:32.873806 (+ 720us) client_negotiation.cc:240] Negotiation successful
0929 18:55:32.874002 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260929 18:55:32.875036 10650 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44247
I20260929 18:55:32.875180 10650 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:32.875419 10650 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
W20260929 18:55:32.876101 10650 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:44247 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:55:32.876233 10650 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:32.876358 10650 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:32.876716 10650 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:32.876842 10650 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
W20260929 18:55:32.877175 10650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:55:32.971362 10653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:32.971639 10653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:32.971676 10653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:32.973380 10653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:32.973445 10653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:32.973485 10653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:55:32.975703 10653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.71.148:36687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:34029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=46617
--tserver_master_addrs=127.29.71.190:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:32.975979 10653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:32.976272 10653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:32.982873 10661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.983165 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:32.985339 10658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:32.985857 10653 server_base.cc:1034] running on GCE node
I20260929 18:55:32.986078 10653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:32.986372 10653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:32.987552 10653 hybrid_clock.cc:648] HybridClock initialized: now 1790708132987524 us; error 38 us; skew 500 ppm
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:32.990605 10653 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:55:32.991099 10653 webserver.cc:460] Webserver started at http://127.29.71.131:46617/ using document root <none> and password file <none>
I20260929 18:55:32.991355 10653 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:32.991420 10653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:55:32 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:33.006778 10653 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:55:33.007783 10670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:33.008016 10653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:33.008124 10653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "7bbdfee146de493c9d28786564cb0682"
format_stamp: "Formatted at 2026-09-29 18:55:32 on dist-test-slave-q2kh"
server_key: "30345306c3c3a62ebb6040423b19fef2"
server_key_iv: "58d41ef0b9c27e379c37b4df0b36f4ec"
server_key_version: "kuduclusterkey@0"
I20260929 18:55:33.008623 10653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:33.042954 10653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:33.045298 10653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:33.045468 10653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:33.045737 10653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:33.046104 10653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:33.046177 10653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:33.046216 10653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:33.046237 10653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:33.053054 10653 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:34029
I20260929 18:55:33.053609 10653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:55:33.054874 10783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:34029 every 8 connection(s)
I20260929 18:55:33.063345 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10653
Sep 29 18:55:33 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:33.069584 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:33.055510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44247 (local address 127.29.71.131:43219)
0929 18:55:33.059554 (+ 4044us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:33.059600 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:33.060133 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:33.060422 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:33.060426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:33.060617 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:33.060817 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:33.060821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:33.061809 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:33.061811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:33.062817 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:33.062819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:33.063332 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:33.063333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:33.063542 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:33.064162 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:33.064172 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:33.065853 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:33.067769 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:33.067775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:33.067779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:33.068068 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:33.068289 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:33.068290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:33.068291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:33.068447 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:33.068776 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:33.068782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:33.068993 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 18:55:33.069134 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 18:55:33.069315 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3989,"thread_start_us":82,"threads_started":1}
I20260929 18:55:33.069971 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:33.055803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:43219 (local address 127.29.71.190:44247)
0929 18:55:33.055852 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:33.055854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:33.059711 (+ 3857us) server_negotiation.cc:407] Connection header received
0929 18:55:33.060237 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:33.060238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:33.060252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:33.060351 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:33.060915 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:33.061690 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:33.062900 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:33.063209 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:33.063294 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:33.065965 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:33.065966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:33.065966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:33.065973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:33.067673 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:33.068154 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:33.068155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:33.068155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:33.068214 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:33.068537 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:33.068538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:33.068538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:33.068683 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:33.068745 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:33.069845 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0929 18:55:33.069912 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:55:33.070600 10784 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44247
I20260929 18:55:33.070714 10784 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:33.071193 10784 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
W20260929 18:55:33.071828 10784 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:44247 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:55:33.071898 10784 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:33.071980 10784 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:33.072363 10784 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:33.072440 10784 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
W20260929 18:55:33.072727 10784 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:55:33.073230 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:33.064181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.29.71.190:44247)
0929 18:55:33.064322 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:33.064324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:33.064333 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:55:33.064809 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:33.064810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:33.064824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:33.064897 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:33.065643 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:33.066318 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:33.067783 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:33.068088 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:33.068145 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:33.070079 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:33.070080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:33.070081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:33.070087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:33.071782 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:33.072359 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:33.072360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:33.072360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:33.072414 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:33.072696 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:33.072696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:33.072697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:33.072831 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:33.073006 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:33.073043 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 18:55:33.073122 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
Sep 29 18:55:33 dist-test-slave-q2kh krb5kdc[9322](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708133, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:33.693747 10516 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:33.693928 10516 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:33.694365 10318 ts_manager.cc:194] Registered new tserver with Master: 1c73f0bd4e3f47598c067297f8b8b33c (127.29.71.129:44333)
I20260929 18:55:33.695323 10318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:57877
I20260929 18:55:33.877414 10650 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:33.877586 10650 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:33.878062 10318 ts_manager.cc:194] Registered new tserver with Master: b109470f33b74ee0920218bb1ac9f21c (127.29.71.130:39385)
I20260929 18:55:33.878641 10318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:51965
Sep 29 18:55:33 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708133, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:34.072923 10784 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:34.073105 10784 heartbeater.cc:511] Master 127.29.71.190:44247 requested a full tablet report, sending...
I20260929 18:55:34.073496 10318 ts_manager.cc:194] Registered new tserver with Master: 7bbdfee146de493c9d28786564cb0682 (127.29.71.131:34029)
I20260929 18:55:34.073946 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:55:34.074011 10318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:43219
I20260929 18:55:34.082849 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.075097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.29.71.190:44247)
0929 18:55:34.075222 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:34.075224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:34.075317 (+ 93us) server_negotiation.cc:407] Connection header received
0929 18:55:34.075387 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:34.075388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:34.075401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:34.075470 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:34.076166 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.076940 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.077946 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.078287 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.078364 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.079798 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:34.079800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:34.079800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:34.079808 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:34.081614 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:34.082001 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:34.082002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:34.082003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:34.082069 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:34.082297 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:34.082298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:34.082299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:34.082434 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:34.082539 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:34.082657 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 18:55:34.082743 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260929 18:55:34.085135 10318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:55:34.087601 10318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:55:34.099040 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.090649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:34029 (local address 127.0.0.1:57206)
0929 18:55:34.094263 (+ 3614us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:34.094275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:34.094343 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:34.094619 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:34.094620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:34.094622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:34.094730 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.094732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.096276 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.096277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.097416 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.097418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.098787 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.098788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.098849 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.098853 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:55:34.098902 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:55:34.098943 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3595,"mutex_wait_us":26,"thread_start_us":109,"threads_started":1}
I20260929 18:55:34.099524 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.090996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.29.71.131:34029)
0929 18:55:34.091337 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:34.091342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:34.094410 (+ 3068us) server_negotiation.cc:407] Connection header received
0929 18:55:34.094452 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:34.094454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:34.094470 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:34.094537 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:34.094815 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.096192 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.097515 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.098347 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.098683 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.098749 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:34.099400 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 18:55:34.099442 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":48,"threads_started":1}
I20260929 18:55:34.100564 10718 tablet_service.cc:1467] Processing CreateTablet for tablet 08eff08add33472191256f116ae4aa7d (DEFAULT_TABLE table=test-table [id=9a25085272a84788b36b636e1924649c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:34.100991 10718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08eff08add33472191256f116ae4aa7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:34.099359 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.090648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:39385 (local address 127.0.0.1:46004)
0929 18:55:34.091013 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:34.091054 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:34.091117 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:34.094609 (+ 3492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:34.094613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:34.094615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:34.094728 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.094731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.096032 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.096034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.097398 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.097401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.098641 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.098642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.098725 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.098742 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:55:34.098826 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:55:34.098888 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":5,"thread_start_us":76,"threads_started":1}
I20260929 18:55:34.104803 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.090748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.29.71.130:39385)
0929 18:55:34.094315 (+ 3567us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:34.094320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:34.094334 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:55:34.094377 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:34.094379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:34.094399 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:34.094494 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:34.094841 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.095918 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.097657 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.098457 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.098541 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.098602 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:34.104660 (+ 6058us) server_negotiation.cc:299] Negotiation successful
0929 18:55:34.104719 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3515,"thread_start_us":50,"threads_started":1}
I20260929 18:55:34.105665 10584 tablet_service.cc:1467] Processing CreateTablet for tablet 08eff08add33472191256f116ae4aa7d (DEFAULT_TABLE table=test-table [id=9a25085272a84788b36b636e1924649c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:34.106081 10584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08eff08add33472191256f116ae4aa7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:34.106395 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.091191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.29.71.129:44333)
0929 18:55:34.091389 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:34.091393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:34.097612 (+ 6219us) server_negotiation.cc:407] Connection header received
0929 18:55:34.099123 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:34.099127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:34.099147 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:34.099212 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:34.099609 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.103004 (+ 3395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.105092 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.105844 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.105896 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.105949 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:34.106250 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:55:34.106303 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":54,"threads_started":1}
I20260929 18:55:34.106624 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.090868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:44333 (local address 127.0.0.1:40794)
0929 18:55:34.097536 (+ 6668us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:34.097547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:34.098982 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:34.099457 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:34.099458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:34.099460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:34.099513 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.099514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.103426 (+ 3912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.103427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.104547 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.104549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.106036 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.106037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.106116 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.106122 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:55:34.106507 (+ 385us) client_negotiation.cc:240] Negotiation successful
0929 18:55:34.106559 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6647,"thread_start_us":65,"threads_started":1}
I20260929 18:55:34.107375 10450 tablet_service.cc:1467] Processing CreateTablet for tablet 08eff08add33472191256f116ae4aa7d (DEFAULT_TABLE table=test-table [id=9a25085272a84788b36b636e1924649c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:34.107751 10450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08eff08add33472191256f116ae4aa7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:34.109036 10819 tablet_bootstrap.cc:492] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Bootstrap starting.
I20260929 18:55:34.109771 10819 tablet_bootstrap.cc:654] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:34.110140 10819 log.cc:824] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:34.111472 10821 tablet_bootstrap.cc:492] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c: Bootstrap starting.
I20260929 18:55:34.112176 10821 tablet_bootstrap.cc:654] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:34.112527 10821 log.cc:824] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:34.114331 10818 tablet_bootstrap.cc:492] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Bootstrap starting.
I20260929 18:55:34.114792 10821 tablet_bootstrap.cc:492] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c: No bootstrap required, opened a new log
I20260929 18:55:34.114939 10821 ts_tablet_manager.cc:1397] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:55:34.114951 10819 tablet_bootstrap.cc:492] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: No bootstrap required, opened a new log
I20260929 18:55:34.115049 10818 tablet_bootstrap.cc:654] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:34.115095 10819 ts_tablet_manager.cc:1397] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260929 18:55:34.115397 10818 log.cc:824] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:34.117424 10819 raft_consensus.cc:348] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.117482 10821 raft_consensus.cc:348] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.117596 10819 raft_consensus.cc:374] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:34.117631 10821 raft_consensus.cc:374] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:34.117635 10819 raft_consensus.cc:729] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b109470f33b74ee0920218bb1ac9f21c, State: Initialized, Role: FOLLOWER
I20260929 18:55:34.117659 10821 raft_consensus.cc:729] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c73f0bd4e3f47598c067297f8b8b33c, State: Initialized, Role: FOLLOWER
I20260929 18:55:34.117766 10821 consensus_queue.cc:260] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.117762 10819 consensus_queue.cc:260] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.118019 10821 ts_tablet_manager.cc:1428] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:55:34.118187 10516 heartbeater.cc:503] Master 127.29.71.190:44247 was elected leader, sending a full tablet report...
I20260929 18:55:34.119050 10819 ts_tablet_manager.cc:1428] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260929 18:55:34.119179 10650 heartbeater.cc:503] Master 127.29.71.190:44247 was elected leader, sending a full tablet report...
I20260929 18:55:34.123067 10818 tablet_bootstrap.cc:492] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: No bootstrap required, opened a new log
I20260929 18:55:34.123132 10818 ts_tablet_manager.cc:1397] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260929 18:55:34.124935 10818 raft_consensus.cc:348] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.125077 10818 raft_consensus.cc:374] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:34.125098 10818 raft_consensus.cc:729] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bbdfee146de493c9d28786564cb0682, State: Initialized, Role: FOLLOWER
I20260929 18:55:34.125232 10818 consensus_queue.cc:260] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.125515 10818 ts_tablet_manager.cc:1428] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:55:34.125547 10784 heartbeater.cc:503] Master 127.29.71.190:44247 was elected leader, sending a full tablet report...
W20260929 18:55:34.174175 10517 tablet.cc:2369] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:55:34 dist-test-slave-q2kh krb5kdc[9322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708133, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for HTTP/127.29.71.148@KRBTEST.COM
I20260929 18:55:34.306572 10824 raft_consensus.cc:482] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:34.306753 10824 raft_consensus.cc:504] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.307188 10824 leader_election.cc:288] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b109470f33b74ee0920218bb1ac9f21c (127.29.71.130:39385), 7bbdfee146de493c9d28786564cb0682 (127.29.71.131:34029)
W20260929 18:55:34.310766 10785 tablet.cc:2369] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:55:34.318181 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.307934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:47343 (local address 127.29.71.130:39385)
0929 18:55:34.307974 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:34.307976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:34.314317 (+ 6341us) server_negotiation.cc:407] Connection header received
0929 18:55:34.314430 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:34.314431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:34.314440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:34.314468 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:34.314677 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.315366 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.317153 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.317748 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.317799 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.317809 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:34.318037 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 18:55:34.318083 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:55:34.318461 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.307738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:39385 (local address 127.29.71.129:47343)
0929 18:55:34.314243 (+ 6505us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:34.314256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:34.314355 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:34.314532 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:34.314533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:34.314534 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:34.314602 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.314604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.315475 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.315476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.316595 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.316598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.317882 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.317883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.317952 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.317955 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:55:34.318308 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 18:55:34.318362 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6485,"thread_start_us":86,"threads_started":1}
I20260929 18:55:34.318787 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eff08add33472191256f116ae4aa7d" candidate_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b109470f33b74ee0920218bb1ac9f21c" is_pre_election: true
I20260929 18:55:34.318944 10604 raft_consensus.cc:2454] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c73f0bd4e3f47598c067297f8b8b33c in term 0.
I20260929 18:55:34.319173 10406 leader_election.cc:302] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c73f0bd4e3f47598c067297f8b8b33c, b109470f33b74ee0920218bb1ac9f21c; no voters:
I20260929 18:55:34.319307 10824 raft_consensus.cc:2781] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:55:34.319360 10824 raft_consensus.cc:482] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:55:34.319387 10824 raft_consensus.cc:3037] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:34.319767 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.307527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:32989 (local address 127.29.71.131:34029)
0929 18:55:34.307599 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:34.307601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:34.311356 (+ 3755us) server_negotiation.cc:407] Connection header received
0929 18:55:34.312802 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:34.312803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:34.312816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:34.312850 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:34.313668 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.314467 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.318503 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.319068 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.319119 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.319127 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:34.319657 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 18:55:34.319700 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:55:34.320128 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.307443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:34029 (local address 127.29.71.129:32989)
0929 18:55:34.311251 (+ 3808us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:34.311268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:34.312554 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:34.312930 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:34.312931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:34.312933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:34.313018 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.313020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.314594 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.314595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.315913 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:34.315916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.319500 (+ 3584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.319501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:34.319578 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.319581 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:55:34.319996 (+ 415us) client_negotiation.cc:240] Negotiation successful
0929 18:55:34.320042 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3784,"thread_start_us":86,"threads_started":1}
I20260929 18:55:34.320289 10824 raft_consensus.cc:504] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.320446 10824 leader_election.cc:288] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [CANDIDATE]: Term 1 election: Requested vote from peers b109470f33b74ee0920218bb1ac9f21c (127.29.71.130:39385), 7bbdfee146de493c9d28786564cb0682 (127.29.71.131:34029)
I20260929 18:55:34.320597 10738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eff08add33472191256f116ae4aa7d" candidate_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bbdfee146de493c9d28786564cb0682" is_pre_election: true
I20260929 18:55:34.320636 10604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eff08add33472191256f116ae4aa7d" candidate_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b109470f33b74ee0920218bb1ac9f21c"
I20260929 18:55:34.320705 10604 raft_consensus.cc:3037] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:34.320766 10738 raft_consensus.cc:2454] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c73f0bd4e3f47598c067297f8b8b33c in term 0.
I20260929 18:55:34.321079 10738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08eff08add33472191256f116ae4aa7d" candidate_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bbdfee146de493c9d28786564cb0682"
I20260929 18:55:34.321151 10738 raft_consensus.cc:3037] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:34.321553 10604 raft_consensus.cc:2454] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c73f0bd4e3f47598c067297f8b8b33c in term 1.
I20260929 18:55:34.321715 10406 leader_election.cc:302] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c73f0bd4e3f47598c067297f8b8b33c, b109470f33b74ee0920218bb1ac9f21c; no voters:
I20260929 18:55:34.321797 10824 raft_consensus.cc:2781] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:34.321926 10824 raft_consensus.cc:686] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [term 1 LEADER]: Becoming Leader. State: Replica: 1c73f0bd4e3f47598c067297f8b8b33c, State: Running, Role: LEADER
I20260929 18:55:34.321990 10738 raft_consensus.cc:2454] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c73f0bd4e3f47598c067297f8b8b33c in term 1.
I20260929 18:55:34.322017 10824 consensus_queue.cc:237] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } }
I20260929 18:55:34.322840 10318 catalog_manager.cc:5495] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c73f0bd4e3f47598c067297f8b8b33c (127.29.71.129). New cstate: current_term: 1 leader_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c73f0bd4e3f47598c067297f8b8b33c" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 44333 } health_report { overall_health: HEALTHY } } }
W20260929 18:55:34.362982 10651 tablet.cc:2369] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:55:34.387286 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.382785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.29.71.129:44333)
0929 18:55:34.382831 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:34.382833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:34.382855 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:55:34.382901 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:34.382902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:34.382926 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:34.382998 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:55:34.383191 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.383936 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.385066 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:34.386430 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:34.386504 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:34.386715 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:55:34.386858 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:55:34.386930 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:34.387181 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 18:55:34.387203 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:55:34.398788 10738 raft_consensus.cc:1261] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 1 FOLLOWER]: Refusing update from remote peer 1c73f0bd4e3f47598c067297f8b8b33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:34.399273 10824 consensus_queue.cc:1035] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bbdfee146de493c9d28786564cb0682" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 34029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:34.401249 10604 raft_consensus.cc:1261] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 1 FOLLOWER]: Refusing update from remote peer 1c73f0bd4e3f47598c067297f8b8b33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:34.401669 10824 consensus_queue.cc:1035] T 08eff08add33472191256f116ae4aa7d P 1c73f0bd4e3f47598c067297f8b8b33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b109470f33b74ee0920218bb1ac9f21c" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:34.408106 10834 mvcc.cc:205] Tried to move back new op lower bound from 7334740518480379904 to 7334740518183653376. Current Snapshot: MvccSnapshot[applied={T|T < 7334740518480379904}]
I20260929 18:55:34.408305 10837 mvcc.cc:205] Tried to move back new op lower bound from 7334740518480379904 to 7334740518183653376. Current Snapshot: MvccSnapshot[applied={T|T < 7334740518480379904}]
I20260929 18:55:34.412052 10835 mvcc.cc:205] Tried to move back new op lower bound from 7334740518480379904 to 7334740518183653376. Current Snapshot: MvccSnapshot[applied={T|T < 7334740518480379904}]
I20260929 18:55:34.412834 10318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:55:34.412963 10318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:55:34.413805 10318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db40a02cb3da49b99312cfec6540faa5: Sending DeleteTablet for 3 replicas of tablet 08eff08add33472191256f116ae4aa7d
I20260929 18:55:34.414273 10584 tablet_service.cc:1514] Processing DeleteTablet for tablet 08eff08add33472191256f116ae4aa7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:34 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46004
I20260929 18:55:34.414683 10718 tablet_service.cc:1514] Processing DeleteTablet for tablet 08eff08add33472191256f116ae4aa7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:34 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:57206
I20260929 18:55:34.415099 10450 tablet_service.cc:1514] Processing DeleteTablet for tablet 08eff08add33472191256f116ae4aa7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:34 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:40794
I20260929 18:55:34.415596 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10356
I20260929 18:55:34.418968 10843 tablet_replica.cc:330] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: stopping tablet replica
I20260929 18:55:34.419094 10843 raft_consensus.cc:2229] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:55:34.419226 10843 raft_consensus.cc:2258] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:34.419395 10844 tablet_replica.cc:330] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: stopping tablet replica
I20260929 18:55:34.419507 10844 raft_consensus.cc:2229] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:55:34.419588 10844 raft_consensus.cc:2258] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:34.419852 10843 ts_tablet_manager.cc:1905] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:55:34.421828 10843 ts_tablet_manager.cc:1918] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:55:34.421960 10843 log.cc:1196] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/08eff08add33472191256f116ae4aa7d
I20260929 18:55:34.422330 10843 ts_tablet_manager.cc:1939] T 08eff08add33472191256f116ae4aa7d P b109470f33b74ee0920218bb1ac9f21c: Deleting consensus metadata
I20260929 18:55:34.422863 10305 catalog_manager.cc:4841] TS b109470f33b74ee0920218bb1ac9f21c (127.29.71.130:39385): tablet 08eff08add33472191256f116ae4aa7d (table test-table [id=9a25085272a84788b36b636e1924649c]) successfully deleted
I20260929 18:55:34.431335 10844 ts_tablet_manager.cc:1905] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:55:34.433173 10844 ts_tablet_manager.cc:1918] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:55:34.433252 10844 log.cc:1196] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/08eff08add33472191256f116ae4aa7d
I20260929 18:55:34.433552 10844 ts_tablet_manager.cc:1939] T 08eff08add33472191256f116ae4aa7d P 7bbdfee146de493c9d28786564cb0682: Deleting consensus metadata
W20260929 18:55:34.433584 10302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:44333 (error 108)
I20260929 18:55:34.434021 10303 catalog_manager.cc:4841] TS 7bbdfee146de493c9d28786564cb0682 (127.29.71.131:34029): tablet 08eff08add33472191256f116ae4aa7d (table test-table [id=9a25085272a84788b36b636e1924649c]) successfully deleted
I20260929 18:55:34.434546 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10520
I20260929 18:55:34.442795 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:34.442645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:44333 (local address 127.0.0.1:40812)
0929 18:55:34.442703 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:34.442752 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 18:55:34.443055 10302 catalog_manager.cc:4584] TS 1c73f0bd4e3f47598c067297f8b8b33c (127.29.71.129:44333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08eff08add33472191256f116ae4aa7d: Network error: Client connection negotiation failed: client connection to 127.29.71.129:44333: connect: Connection refused (error 111)
I20260929 18:55:34.443993 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10653
I20260929 18:55:34.460836 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10285
I20260929 18:55:34.466384 29982 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:55:34.811069 29982 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:55:34.811221 29982 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:55:35.155745 29982 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:55:35.155 UTC [9352] LOG: received smart shutdown request
2026-09-29 18:55:35.160 UTC [9352] LOG: background worker "logical replication launcher" (PID 9361) exited with exit code 1
2026-09-29 18:55:35.160 UTC [9356] LOG: shutting down
2026-09-29 18:55:35.342 UTC [9352] LOG: database system is shut down
2026-09-29T18:55:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47828 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:55:35 dist-test-slave-q2kh krb5kdc[10856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:55:35 dist-test-slave-q2kh krb5kdc[10856](info): set up 2 sockets
Sep 29 18:55:35 dist-test-slave-q2kh krb5kdc[10856](info): commencing operation
krb5kdc: starting...
W20260929 18:55:36.506069 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:55:36 dist-test-slave-q2kh krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:55:36.626056 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45359
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:45359,127.29.71.189:40993,127.29.71.188:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:36.707613 10869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:36.707831 10869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:36.707854 10869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:36.709071 10869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:36.709115 10869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:36.709131 10869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:36.709142 10869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:36.709182 10869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:36.710767 10869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:45359,127.29.71.189:40993,127.29.71.188:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:36.711053 10869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:36.711300 10869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:36.716200 10874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:36.716230 10875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:36.716270 10869 server_base.cc:1034] running on GCE node
W20260929 18:55:36.716221 10877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:36.716739 10869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:36.717034 10869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:36.718227 10869 hybrid_clock.cc:648] HybridClock initialized: now 1790708136718206 us; error 47 us; skew 500 ppm
Sep 29 18:55:36 dist-test-slave-q2kh krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:36.724195 10869 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:36.724627 10869 webserver.cc:460] Webserver started at http://127.29.71.190:36317/ using document root <none> and password file <none>
I20260929 18:55:36.724855 10869 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:36.724923 10869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:36.725075 10869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:36.726243 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "44ce352e327c4cfaaa81d541e130f784"
format_stamp: "Formatted at 2026-09-29 18:55:36 on dist-test-slave-q2kh"
server_key: "72e92223b4cdd0402d10a8ff65624839"
server_key_iv: "1097f56eeeef10fd34c5fe7de387bad2"
server_key_version: "encryptionkey@0"
I20260929 18:55:36.726604 10869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "44ce352e327c4cfaaa81d541e130f784"
format_stamp: "Formatted at 2026-09-29 18:55:36 on dist-test-slave-q2kh"
server_key: "72e92223b4cdd0402d10a8ff65624839"
server_key_iv: "1097f56eeeef10fd34c5fe7de387bad2"
server_key_version: "encryptionkey@0"
I20260929 18:55:36.727983 10869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:55:36.728698 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:36.728912 10869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:55:36.729012 10869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "44ce352e327c4cfaaa81d541e130f784"
format_stamp: "Formatted at 2026-09-29 18:55:36 on dist-test-slave-q2kh"
server_key: "72e92223b4cdd0402d10a8ff65624839"
server_key_iv: "1097f56eeeef10fd34c5fe7de387bad2"
server_key_version: "encryptionkey@0"
I20260929 18:55:36.729090 10869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:36.757874 10869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:36.760172 10869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:36.760354 10869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:36.764482 10869 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:45359
I20260929 18:55:36.764599 10936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:45359 every 8 connection(s)
I20260929 18:55:36.764919 10869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:36.765865 10937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:36.767479 10937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:55:36.771097 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10869
I20260929 18:55:36.771226 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:55:36.771430 29982 external_mini_cluster.cc:1428] Setting key 58c308099ee7fa6a073a82d54f486213
Sep 29 18:55:36 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:36.777871 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.768235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.29.71.190:45359)
0929 18:55:36.768419 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:36.768429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:36.768557 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:55:36.769199 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:36.769201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:36.769218 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:36.769316 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:36.769892 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.770775 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.771862 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.772167 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.772293 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.774456 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:36.774464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:36.774464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:36.774472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:36.776168 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.776621 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.776624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.776624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.776708 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.776938 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.776939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.776939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.777164 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:36.777266 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:36.777454 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 18:55:36.777617 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260929 18:55:36.777877 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.768164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45359 (local address 127.0.0.1:48632)
0929 18:55:36.768408 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.768454 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:36.769025 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:36.769323 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:36.769326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:36.769616 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:36.769768 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.769773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.770885 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.770886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.771763 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.771766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.772263 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.772264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.772326 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.772882 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:36.772891 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:36.774373 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:36.776304 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.776310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.776312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.776507 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.776786 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.776787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.776787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.776872 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.777243 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:36.777247 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:36.777323 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:55:36.777430 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:55:36.777665 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":96,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.29.71.189@KRBTEST.COM; defaulting to no policy
I20260929 18:55:36.778599 10937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:55:36.778975 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.778864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41606)
0929 18:55:36.778915 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.778956 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:55:36.779094 10887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
I20260929 18:55:36.779464 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.779364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41618)
0929 18:55:36.779415 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.779440 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:55:36.779567 10937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:40993: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab.
Entry for principal kudu/127.29.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab.
I20260929 18:55:36.797510 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:40993
--webserver_interface=127.29.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab
--principal=kudu/127.29.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:45359,127.29.71.189:40993,127.29.71.188:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:36.828809 10937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } attempt: 1
I20260929 18:55:36.829396 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.829248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41626)
0929 18:55:36.829317 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.829365 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 18:55:36.829843 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.829707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41640)
0929 18:55:36.829753 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.829824 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 18:55:36.829964 10937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:40993: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
W20260929 18:55:36.879338 10944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:36.879578 10944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:36.879614 10944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:36.880713 10944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:36.880779 10944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:36.880803 10944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:36.880824 10944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:36.880882 10944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:36.882391 10944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:45359,127.29.71.189:40993,127.29.71.188:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab
--principal=kudu/127.29.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:40993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:36.882656 10944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:36.882884 10944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:36.887720 10950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:36.887701 10949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:36.887751 10944 server_base.cc:1034] running on GCE node
W20260929 18:55:36.887699 10952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:36.888185 10944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:36.888495 10944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:36.889643 10944 hybrid_clock.cc:648] HybridClock initialized: now 1790708136889626 us; error 39 us; skew 500 ppm
Sep 29 18:55:36 dist-test-slave-q2kh krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:36.892400 10944 init.cc:377] Logged in from keytab as kudu/127.29.71.189@KRBTEST.COM (short username kudu)
I20260929 18:55:36.892822 10944 webserver.cc:460] Webserver started at http://127.29.71.189:37945/ using document root <none> and password file <none>
I20260929 18:55:36.893061 10944 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:36.893106 10944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:36.893270 10944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:36.894457 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data/instance:
uuid: "3ef416cb7040477ca172168bbd199db4"
format_stamp: "Formatted at 2026-09-29 18:55:36 on dist-test-slave-q2kh"
server_key: "7b6b68685aa41e1605a21b3c2a45683d"
server_key_iv: "b69d2143dc5e353664e0ffe6bb36fded"
server_key_version: "encryptionkey@0"
I20260929 18:55:36.894814 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/wal/instance:
uuid: "3ef416cb7040477ca172168bbd199db4"
format_stamp: "Formatted at 2026-09-29 18:55:36 on dist-test-slave-q2kh"
server_key: "7b6b68685aa41e1605a21b3c2a45683d"
server_key_iv: "b69d2143dc5e353664e0ffe6bb36fded"
server_key_version: "encryptionkey@0"
I20260929 18:55:36.896112 10944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:36.896890 10959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:36.897171 10944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:36.897301 10944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/wal
uuid: "3ef416cb7040477ca172168bbd199db4"
format_stamp: "Formatted at 2026-09-29 18:55:36 on dist-test-slave-q2kh"
server_key: "7b6b68685aa41e1605a21b3c2a45683d"
server_key_iv: "b69d2143dc5e353664e0ffe6bb36fded"
server_key_version: "encryptionkey@0"
I20260929 18:55:36.897401 10944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:36.898101 10937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } attempt: 2
I20260929 18:55:36.898559 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.898440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41648)
0929 18:55:36.898499 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.898533 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:55:36.898904 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.898816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41654)
0929 18:55:36.898862 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.898888 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:55:36.898991 10937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:40993: Network error: Client connection negotiation failed: client connection to 127.29.71.189:40993: connect: Connection refused (error 111)
I20260929 18:55:36.909301 10944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:36.911522 10944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:36.911689 10944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:36.915935 10944 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.189:40993
I20260929 18:55:36.915967 11011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.189:40993 every 8 connection(s)
I20260929 18:55:36.916318 10944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
I20260929 18:55:36.917196 11012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:36.918908 11012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:55:36.922434 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 10944
I20260929 18:55:36.922557 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-1/wal/instance
I20260929 18:55:36.922749 29982 external_mini_cluster.cc:1428] Setting key 51414242708e343c2f883116006f4217
Sep 29 18:55:36 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:36.928608 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.919648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.29.71.190:45359)
0929 18:55:36.919715 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:36.919718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:36.919910 (+ 192us) server_negotiation.cc:407] Connection header received
0929 18:55:36.920407 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:36.920408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:36.920421 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:36.920476 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:36.921097 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.921591 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.922734 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.922990 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.923064 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.925464 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:36.925466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:36.925466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:36.925475 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:36.927186 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.927581 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.927583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.927583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.927652 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.927885 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.927885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.927886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.928049 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:36.928130 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:36.928386 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 18:55:36.928488 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:55:36.928618 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.919639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45359 (local address 127.0.0.1:48644)
0929 18:55:36.919793 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.919827 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:36.920291 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:36.920549 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:36.920552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:36.920796 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:36.920993 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.920997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.921680 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.921681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.922642 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.922645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.923070 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.923071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.923200 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.923719 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:36.923727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:36.925367 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:36.927273 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.927278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.927280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.927503 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.927725 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.927725 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.927726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.927819 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.928122 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:36.928126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:36.928244 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 18:55:36.928330 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:55:36.928456 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260929 18:55:36.929299 11012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.71.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.188@KRBTEST.COM" created.
Sep 29 18:55:36 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:55:36.937775 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.929580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41662)
0929 18:55:36.929627 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.929638 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:36.929679 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:36.929959 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:36.929960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:36.930070 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:36.930117 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.930118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.931459 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.931460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.932016 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.932018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.932492 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.932493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.932547 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.932890 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:36.932894 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:36.934820 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:36.936613 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.936614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.936615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.936840 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.937049 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.937050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.937050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.937119 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.937376 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:36.937377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:36.937425 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:55:36.937501 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:55:36.937615 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:55:36.937788 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.929635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.29.71.189:40993)
0929 18:55:36.929792 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:36.929797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:36.929820 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:55:36.929843 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:36.929846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:36.929861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:36.929945 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:36.930212 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.931372 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.932080 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.932427 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.932483 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.934934 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:36.934942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:36.934943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:36.934949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:36.936532 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.936918 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.936921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.936922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.936986 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.937178 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.937179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.937179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.937306 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:36.937365 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:36.937534 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 18:55:36.937704 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260929 18:55:36.938740 11012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:55:36.939134 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.938994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33772)
0929 18:55:36.939048 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.939103 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:55:36.939271 10961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
I20260929 18:55:36.939718 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.939634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33786)
0929 18:55:36.939666 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.939696 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 18:55:36.939855 11012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:32829: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab.
Entry for principal kudu/127.29.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab.
I20260929 18:55:36.949934 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:32829
--webserver_interface=127.29.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab
--principal=kudu/127.29.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:45359,127.29.71.189:40993,127.29.71.188:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:36.989087 11012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } attempt: 1
I20260929 18:55:36.989653 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.989496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33788)
0929 18:55:36.989574 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.989620 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:55:36.990190 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.989971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33798)
0929 18:55:36.990052 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.990087 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:55:36.990185 10937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } attempt: 3
W20260929 18:55:36.990307 11012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:32829: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Sep 29 18:55:36 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:55:36.997462 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.990464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41666)
0929 18:55:36.990523 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.990536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:36.990593 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:36.990769 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:36.990770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:36.990969 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:36.991043 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.991046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.991843 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.991844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.992400 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:36.992401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.992839 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.992840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:36.992930 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.993418 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:36.993426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:36.994718 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:36.996371 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.996373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.996373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.996631 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.996882 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:36.996882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:36.996883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:36.996942 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:36.997204 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:36.997205 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:36.997243 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:55:36.997311 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:55:36.997388 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:55:36.997515 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.990514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.29.71.189:40993)
0929 18:55:36.990555 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:36.990557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:36.990572 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:55:36.990696 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:36.990697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:36.990707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:36.990759 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:36.991156 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.991727 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.992493 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:36.992738 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:36.992838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:36.994817 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:36.994819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:36.994820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:36.994827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:36.996242 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.996718 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.996720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.996720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.996790 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:36.997006 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:36.997007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:36.997007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:36.997139 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:36.997206 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:36.997350 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:55:36.997435 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:55:36.998281 10937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:55:36.998624 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.998527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33802)
0929 18:55:36.998576 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.998604 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:55:36.998989 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:36.998892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33816)
0929 18:55:36.998940 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:36.998964 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 18:55:36.999074 10937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:32829: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
I20260929 18:55:37.030231 10937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } attempt: 1
I20260929 18:55:37.030710 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.030575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33832)
0929 18:55:37.030640 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.030682 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:55:37.031205 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.031078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33848)
0929 18:55:37.031150 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.031181 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 18:55:37.031340 10937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:32829: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
W20260929 18:55:37.033515 11019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:37.033730 11019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:37.033779 11019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:37.035013 11019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:37.035076 11019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:37.035099 11019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:37.035120 11019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:37.035151 11019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:37.036682 11019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:45359,127.29.71.189:40993,127.29.71.188:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab
--principal=kudu/127.29.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:37.036949 11019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:37.037174 11019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:37.042323 11024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:37.042424 11019 server_base.cc:1034] running on GCE node
W20260929 18:55:37.042330 11025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:37.042332 11027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:37.042834 11019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:37.043102 11019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:37.044250 11019 hybrid_clock.cc:648] HybridClock initialized: now 1790708137044223 us; error 43 us; skew 500 ppm
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708137, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:37.046852 11019 init.cc:377] Logged in from keytab as kudu/127.29.71.188@KRBTEST.COM (short username kudu)
I20260929 18:55:37.047282 11019 webserver.cc:460] Webserver started at http://127.29.71.188:45059/ using document root <none> and password file <none>
I20260929 18:55:37.047495 11019 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:37.047540 11019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:37.047631 11019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:37.048678 11019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data/instance:
uuid: "52b71a0694c547f492d6983c0d4094d7"
format_stamp: "Formatted at 2026-09-29 18:55:37 on dist-test-slave-q2kh"
server_key: "069a6f3d7153448eb0274fad54f65e7f"
server_key_iv: "7a8316ed180d6a9ff2dfeaabbe8ec81c"
server_key_version: "encryptionkey@0"
I20260929 18:55:37.049010 11019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/wal/instance:
uuid: "52b71a0694c547f492d6983c0d4094d7"
format_stamp: "Formatted at 2026-09-29 18:55:37 on dist-test-slave-q2kh"
server_key: "069a6f3d7153448eb0274fad54f65e7f"
server_key_iv: "7a8316ed180d6a9ff2dfeaabbe8ec81c"
server_key_version: "encryptionkey@0"
I20260929 18:55:37.050490 11019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:55:37.051186 11034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:37.051396 11019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:55:37.051479 11019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/wal
uuid: "52b71a0694c547f492d6983c0d4094d7"
format_stamp: "Formatted at 2026-09-29 18:55:37 on dist-test-slave-q2kh"
server_key: "069a6f3d7153448eb0274fad54f65e7f"
server_key_iv: "7a8316ed180d6a9ff2dfeaabbe8ec81c"
server_key_version: "encryptionkey@0"
I20260929 18:55:37.051537 11019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:37.058444 11012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } attempt: 2
I20260929 18:55:37.058838 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.058715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33854)
0929 18:55:37.058772 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.058812 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:55:37.059165 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.059066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33858)
0929 18:55:37.059112 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.059136 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:55:37.059258 11012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:32829: Network error: Client connection negotiation failed: client connection to 127.29.71.188:32829: connect: Connection refused (error 111)
I20260929 18:55:37.070809 11019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:37.073093 11019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:37.073251 11019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:37.077215 11019 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.188:32829
I20260929 18:55:37.077270 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.188:32829 every 8 connection(s)
I20260929 18:55:37.077633 11019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
I20260929 18:55:37.078620 11087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:37.080336 11087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:55:37.084889 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11019
I20260929 18:55:37.085024 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790707954762284-29982-0/minicluster-data/master-2/wal/instance
I20260929 18:55:37.085284 29982 external_mini_cluster.cc:1428] Setting key 2cb045175b796ea49a0d65877edc7455
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708137, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:37.091238 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.081106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.29.71.190:45359)
0929 18:55:37.081184 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.081187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.081394 (+ 207us) server_negotiation.cc:407] Connection header received
0929 18:55:37.081921 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.081923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.081932 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.081990 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.082788 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.083266 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.084500 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.084692 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.084762 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.087525 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.087527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.087527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.087535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.089408 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.090051 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.090053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.090053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.090128 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.090476 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.090477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.090477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.090614 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.090708 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.091031 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.091131 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:55:37.091424 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.081086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45359 (local address 127.0.0.1:48660)
0929 18:55:37.081266 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.081306 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:37.081753 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:37.082098 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:37.082102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:37.082454 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:37.082662 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.082667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.083369 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.083370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.084417 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.084420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.084815 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.084817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.084997 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.085590 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:37.085601 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:37.087437 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:37.089528 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.089536 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.089538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.089937 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.090252 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.090253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.090253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.090376 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.090698 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:37.090705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:37.090864 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 18:55:37.091007 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 18:55:37.091166 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:55:37.092181 11087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:55:37.095573 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.085731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.29.71.188:32829)
0929 18:55:37.085866 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.085870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.086018 (+ 148us) server_negotiation.cc:407] Connection header received
0929 18:55:37.086094 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.086095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.086113 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.086243 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.086993 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.087868 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.088906 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.089449 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.089521 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.092214 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.092217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.092218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.092223 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.094089 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.094463 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.094466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.094466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.094528 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.094712 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.094713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.094713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.094882 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.095036 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.095360 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.095450 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708137, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:55:37.100355 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.092488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:40993 (local address 127.0.0.1:41670)
0929 18:55:37.092581 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.092593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:37.092632 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:37.092787 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:37.092787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:37.092923 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:37.092966 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.092967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.093934 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.093935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.094473 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.094475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.094957 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.094958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.095018 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.095390 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:37.095395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:37.096821 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:37.099140 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.099141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.099142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.099417 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.099678 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.099679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.099679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.099742 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.100059 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:37.100060 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:37.100099 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:55:37.100167 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:55:37.100243 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 18:55:37.100452 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.092533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.29.71.189:40993)
0929 18:55:37.092620 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.092623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.092634 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:55:37.092694 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.092696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.092709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.092780 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.093044 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.093842 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.094534 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.094884 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.094956 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.096910 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.096912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.096912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.096921 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.099019 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.099518 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.099519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.099519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.099598 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.099827 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.099828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.099828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.099978 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.100061 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.100217 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.100316 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:55:37.100965 11087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708137, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:55:37.105532 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.096418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.29.71.190:45359)
0929 18:55:37.096459 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.096462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.096480 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:55:37.096503 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.096504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.096528 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.096587 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.097190 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.097621 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.098757 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.098994 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.099069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.101943 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.101944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.101944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.101951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.103646 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.104118 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.104119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.104120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.104181 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.104415 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.104416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.104416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.104528 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.105294 (+ 766us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.105334 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.105408 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:55:37.106465 10937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } attempt: 2
I20260929 18:55:37.108182 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.101270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33864)
0929 18:55:37.101326 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.101338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:37.101381 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:37.101557 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:37.101557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:37.101748 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:37.101791 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.101793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.102584 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.102585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.103086 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.103088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.103499 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.103500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.103552 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.103965 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:37.103970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:37.105193 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:37.107157 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.107158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.107159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.107352 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.107557 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.107557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.107558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.107610 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.107909 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:37.107910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:37.107949 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 18:55:37.108013 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:55:37.108094 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 18:55:37.108490 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.101329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.29.71.188:32829)
0929 18:55:37.101383 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.101385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.101394 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:55:37.101472 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.101473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.101485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.101548 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.101876 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.102499 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.103182 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.103410 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.103499 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.105282 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.105283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.105283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.105290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.107075 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.107437 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.107437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.107438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.107487 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.107682 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.107683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.107684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.107790 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.107915 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.108348 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.108414 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:55:37.110389 11087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: Bootstrap starting.
I20260929 18:55:37.111057 11087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: Neither blocks nor log segments found. Creating new log.
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:55:37.111377 11087 log.cc:824] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:55:37.112922 11087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: No bootstrap required, opened a new log
I20260929 18:55:37.114769 11087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.114931 11087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:37.114981 11087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b71a0694c547f492d6983c0d4094d7, State: Initialized, Role: FOLLOWER
I20260929 18:55:37.115157 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.106722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33876)
0929 18:55:37.106975 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.106986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:37.107029 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:37.107193 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:37.107195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:37.107348 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:37.107384 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.107386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.108579 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.108580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.109300 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.109302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.109639 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.109641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.109700 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.110134 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:37.110140 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:37.111911 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:37.114024 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.114025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.114025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.114267 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.114492 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.114493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.114493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.114549 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.114841 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:37.114842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:37.114884 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:55:37.114969 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:55:37.115058 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260929 18:55:37.115116 11087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.115391 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.106380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.29.71.189:40993)
0929 18:55:37.106420 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.106422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.106431 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:55:37.106503 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.106504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.106514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.106580 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.107202 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.107920 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.108813 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.109112 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.109180 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.112046 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.112048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.112048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.112055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.113853 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.114253 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.114254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.114254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.114310 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.114534 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.114535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.114536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.114667 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.114852 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.115215 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.115306 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:55:37.115463 11087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:37.115490 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.106798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.29.71.188:32829)
0929 18:55:37.107006 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.107008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.107017 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:55:37.107081 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.107082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.107092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.107455 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.107498 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.108479 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.109379 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.109569 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.109636 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.112038 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.112039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.112040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.112046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.113964 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.114361 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.114362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.114363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.114416 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.114624 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.114625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.114626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.114756 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.114828 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.115284 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.115375 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":127,"threads_started":1}
I20260929 18:55:37.115916 11094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.116074 11094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:55:37.116879 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:55:37.117053 10937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784: Bootstrap starting.
I20260929 18:55:37.117786 10937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:37.118100 10937 log.cc:824] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784: Log is configured to *not* fsync() on all Append() calls
W20260929 18:55:37.117259 11105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:55:37.118458 11105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:55:37.118875 10937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784: No bootstrap required, opened a new log
I20260929 18:55:37.120884 10937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.121028 10937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:37.121071 10937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ce352e327c4cfaaa81d541e130f784, State: Initialized, Role: FOLLOWER
I20260929 18:55:37.121192 10937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.119233 11019 master_runner.cc:186] Error getting master registration for 127.29.71.190:45359: OK, instance_id { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" instance_seqno: 1790708136763306 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:55:37.121481 11019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:55:37.121492 10937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:55:37.123448 11126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:55:37.123512 11126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:55:37.123781 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.123875 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:55:37.124085 10869 master_runner.cc:186] Error getting master registration for 127.29.71.189:40993: OK, instance_id { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" instance_seqno: 1790708136914670 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:55:37.126857 10869 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:55:37.127240 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.118266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.29.71.190:45359)
0929 18:55:37.118306 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.118308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.119007 (+ 699us) server_negotiation.cc:407] Connection header received
0929 18:55:37.119107 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.119108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.119117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.119168 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.119910 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.120588 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.121590 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.121857 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.121945 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.124062 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.124063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.124064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.124071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.125985 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.126414 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.126415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.126416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.126470 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.126654 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.126655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.126656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.126765 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.126850 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.127063 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.127139 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:55:37.131691 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.122201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.29.71.188:32829)
0929 18:55:37.122241 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.122243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.122251 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:55:37.122269 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.122270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.122281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.122343 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.124360 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.124893 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.126906 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.127222 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.127266 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.128725 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.128726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.128727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.128733 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.130425 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.130808 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.130809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.130810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.130877 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.131085 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.131086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.131087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.131201 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.131286 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.131498 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.131592 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:55:37.135378 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.118804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.29.71.189:40993)
0929 18:55:37.118851 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.118853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.124050 (+ 5197us) server_negotiation.cc:407] Connection header received
0929 18:55:37.127865 (+ 3815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.127867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.127879 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.127911 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.128548 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.129823 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.130936 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.131252 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.131292 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.132836 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.132837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.132837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.132844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.134274 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.134612 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.134613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.134613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.134667 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.134870 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.134871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.134872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.134982 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.135101 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.135202 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.135274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:55:37.150394 11012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } attempt: 3
Sep 29 18:55:37 dist-test-slave-q2kh krb5kdc[10856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:55:37.169755 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.150826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:32829 (local address 127.0.0.1:33900)
0929 18:55:37.151015 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:37.151027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:37.151073 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:37.151283 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:37.151284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:37.152638 (+ 1354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:37.152697 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.152700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.153597 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.153598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.154772 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:37.154775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.155675 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.155676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:37.155743 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.163148 (+ 7405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:37.163154 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:37.165055 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:37.168466 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.168467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.168468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.168717 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.168974 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:37.168975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:37.168975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:37.169038 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:37.169369 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:37.169369 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:37.169428 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 18:55:37.169512 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:55:37.169634 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 18:55:37.170173 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:37.150936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.29.71.188:32829)
0929 18:55:37.151144 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:37.151146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:37.151153 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:55:37.151177 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:37.151178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:37.151189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:37.151235 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:37.152812 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.153493 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.154991 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:37.155538 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:37.155608 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:37.165559 (+ 9951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:37.165561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:37.165561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:37.165571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:37.168357 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.168828 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.168830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.168831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.168901 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:37.169110 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:37.169111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:37.169111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:37.169271 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:37.169338 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:37.169985 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 18:55:37.170064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260929 18:55:37.172641 11115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:37.172751 11115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.173018 11115 leader_election.cc:288] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ef416cb7040477ca172168bbd199db4 (127.29.71.189:40993), 52b71a0694c547f492d6983c0d4094d7 (127.29.71.188:32829)
I20260929 18:55:37.173312 11012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: Bootstrap starting.
I20260929 18:55:37.173383 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44ce352e327c4cfaaa81d541e130f784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b71a0694c547f492d6983c0d4094d7" is_pre_election: true
I20260929 18:55:37.173547 11061 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44ce352e327c4cfaaa81d541e130f784 in term 0.
I20260929 18:55:37.173766 10886 leader_election.cc:302] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ce352e327c4cfaaa81d541e130f784, 52b71a0694c547f492d6983c0d4094d7; no voters:
I20260929 18:55:37.173890 11115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:55:37.173930 11115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:55:37.173952 11115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:37.174135 11012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:37.174506 11012 log.cc:824] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:37.174821 11115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.174944 11115 leader_election.cc:288] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [CANDIDATE]: Term 1 election: Requested vote from peers 3ef416cb7040477ca172168bbd199db4 (127.29.71.189:40993), 52b71a0694c547f492d6983c0d4094d7 (127.29.71.188:32829)
I20260929 18:55:37.175145 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44ce352e327c4cfaaa81d541e130f784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b71a0694c547f492d6983c0d4094d7"
I20260929 18:55:37.175218 11061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:37.175444 11012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: No bootstrap required, opened a new log
I20260929 18:55:37.176074 11061 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44ce352e327c4cfaaa81d541e130f784 in term 1.
I20260929 18:55:37.176231 10886 leader_election.cc:302] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ce352e327c4cfaaa81d541e130f784, 52b71a0694c547f492d6983c0d4094d7; no voters:
I20260929 18:55:37.176317 11115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:37.176470 11115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [term 1 LEADER]: Becoming Leader. State: Replica: 44ce352e327c4cfaaa81d541e130f784, State: Running, Role: LEADER
I20260929 18:55:37.176581 11115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.176998 11129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ce352e327c4cfaaa81d541e130f784. Latest consensus state: current_term: 1 leader_uuid: "44ce352e327c4cfaaa81d541e130f784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.177233 11129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:37.177977 11012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.178112 11012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:37.178171 11012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef416cb7040477ca172168bbd199db4, State: Initialized, Role: FOLLOWER
I20260929 18:55:37.178315 11012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } }
I20260929 18:55:37.178645 11012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:37.178684 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44ce352e327c4cfaaa81d541e130f784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef416cb7040477ca172168bbd199db4"
I20260929 18:55:37.178762 10985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:37.179759 10985 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44ce352e327c4cfaaa81d541e130f784 in term 1.
I20260929 18:55:37.180495 11131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:37.178617 11132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.180696 11131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:55:37.180773 11132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:55:37.179895 10986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44ce352e327c4cfaaa81d541e130f784" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef416cb7040477ca172168bbd199db4" is_pre_election: true
I20260929 18:55:37.180948 10986 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 44ce352e327c4cfaaa81d541e130f784 in term 1. Re-sending same reply.
I20260929 18:55:37.182317 11061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [term 1 FOLLOWER]: Refusing update from remote peer 44ce352e327c4cfaaa81d541e130f784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:37.182678 11129 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:37.184741 11094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ce352e327c4cfaaa81d541e130f784. Latest consensus state: current_term: 1 leader_uuid: "44ce352e327c4cfaaa81d541e130f784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.184849 11094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:55:37.185334 10986 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [term 1 FOLLOWER]: Refusing update from remote peer 44ce352e327c4cfaaa81d541e130f784: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:37.185621 11129 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:37.186707 11132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ce352e327c4cfaaa81d541e130f784. Latest consensus state: current_term: 1 leader_uuid: "44ce352e327c4cfaaa81d541e130f784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.186792 11132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:55:37.188853 11129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44ce352e327c4cfaaa81d541e130f784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.188954 11129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:37.189317 11132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44ce352e327c4cfaaa81d541e130f784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.189406 11132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:55:37.190204 11131 catalog_manager.cc:1347] Generated new cluster ID: 9be3abfbac1e48578b2f77112138f577
I20260929 18:55:37.190303 11131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260929 18:55:37.190910 11143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:55:37.190963 11143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:55:37.191749 11129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44ce352e327c4cfaaa81d541e130f784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.191854 11129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:37.192410 11094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44ce352e327c4cfaaa81d541e130f784" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ce352e327c4cfaaa81d541e130f784" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45359 } } peers { permanent_uuid: "3ef416cb7040477ca172168bbd199db4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 40993 } } peers { permanent_uuid: "52b71a0694c547f492d6983c0d4094d7" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 32829 } } }
I20260929 18:55:37.192510 11094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:55:37.192610 11145 mvcc.cc:205] Tried to move back new op lower bound from 7334740529896611840 to 7334740529875763200. Current Snapshot: MvccSnapshot[applied={T|T < 7334740529896611840}]
I20260929 18:55:37.200320 11131 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:55:37.201236 11131 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:55:37.210798 11131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ce352e327c4cfaaa81d541e130f784: Generated new TSK 0
I20260929 18:55:37.211109 11131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:55:38.118947 11105 catalog_manager.cc:1259] Loaded cluster ID: 9be3abfbac1e48578b2f77112138f577
I20260929 18:55:38.119042 11105 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: loading cluster ID for follower catalog manager: success
I20260929 18:55:38.119840 11105 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: acquiring CA information for follower catalog manager: success
I20260929 18:55:38.120136 11105 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52b71a0694c547f492d6983c0d4094d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:55:38.191430 11143 catalog_manager.cc:1259] Loaded cluster ID: 9be3abfbac1e48578b2f77112138f577
I20260929 18:55:38.191531 11143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: loading cluster ID for follower catalog manager: success
I20260929 18:55:38.192344 11143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: acquiring CA information for follower catalog manager: success
I20260929 18:55:38.192669 11143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ef416cb7040477ca172168bbd199db4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:55:38.264276 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10869
I20260929 18:55:38.269968 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 10944
I20260929 18:55:38.275218 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11019
2026-09-29T18:55:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2823 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:55:38 dist-test-slave-q2kh krb5kdc[11163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:55:38 dist-test-slave-q2kh krb5kdc[11163](info): set up 2 sockets
Sep 29 18:55:38 dist-test-slave-q2kh krb5kdc[11163](info): commencing operation
krb5kdc: starting...
W20260929 18:55:39.366295 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:55:39 dist-test-slave-q2kh krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:55:39.486316 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41519
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:41519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:39.575553 11176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:39.575817 11176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:39.575853 11176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:39.577122 11176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:39.577165 11176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:39.577179 11176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:39.577194 11176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:39.577206 11176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:39.578682 11176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:41519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:39.578871 11176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:39.579032 11176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:39.583662 11184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:39.583710 11181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:39.583913 11176 server_base.cc:1034] running on GCE node
W20260929 18:55:39.583669 11182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:39.584179 11176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:39.584442 11176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:39.585584 11176 hybrid_clock.cc:648] HybridClock initialized: now 1790708139585577 us; error 33 us; skew 500 ppm
Sep 29 18:55:39 dist-test-slave-q2kh krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708139, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:39.588070 11176 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:39.588544 11176 webserver.cc:460] Webserver started at http://127.29.71.190:39197/ using document root <none> and password file <none>
I20260929 18:55:39.588761 11176 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:39.588828 11176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:39.588980 11176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:39.589979 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "23aca554ceda4118a6c26d81992d8ed7"
format_stamp: "Formatted at 2026-09-29 18:55:39 on dist-test-slave-q2kh"
server_key: "e7741331cf2f6921cf18c5642ac8314b"
server_key_iv: "25e6ce15aa823d2d76048cf603cf6b1d"
server_key_version: "encryptionkey@0"
I20260929 18:55:39.590356 11176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "23aca554ceda4118a6c26d81992d8ed7"
format_stamp: "Formatted at 2026-09-29 18:55:39 on dist-test-slave-q2kh"
server_key: "e7741331cf2f6921cf18c5642ac8314b"
server_key_iv: "25e6ce15aa823d2d76048cf603cf6b1d"
server_key_version: "encryptionkey@0"
I20260929 18:55:39.591632 11176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:55:39.592376 11191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:39.592561 11176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:55:39.592658 11176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "23aca554ceda4118a6c26d81992d8ed7"
format_stamp: "Formatted at 2026-09-29 18:55:39 on dist-test-slave-q2kh"
server_key: "e7741331cf2f6921cf18c5642ac8314b"
server_key_iv: "25e6ce15aa823d2d76048cf603cf6b1d"
server_key_version: "encryptionkey@0"
I20260929 18:55:39.592738 11176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:39.605769 11176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:39.607919 11176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:39.608083 11176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:39.612231 11176 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:41519
I20260929 18:55:39.612275 11243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:41519 every 8 connection(s)
I20260929 18:55:39.612643 11176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:39.613539 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:39.615882 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7: Bootstrap starting.
I20260929 18:55:39.616454 11244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:39.616751 11244 log.cc:824] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:39.617442 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7: No bootstrap required, opened a new log
I20260929 18:55:39.619076 11244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aca554ceda4118a6c26d81992d8ed7" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41519 } }
I20260929 18:55:39.619210 11244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:39.619232 11244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aca554ceda4118a6c26d81992d8ed7, State: Initialized, Role: FOLLOWER
I20260929 18:55:39.619411 11244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aca554ceda4118a6c26d81992d8ed7" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41519 } }
I20260929 18:55:39.619474 11244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:55:39.619495 11244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:55:39.619562 11244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:39.620066 11244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aca554ceda4118a6c26d81992d8ed7" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41519 } }
I20260929 18:55:39.620172 11244 leader_election.cc:302] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23aca554ceda4118a6c26d81992d8ed7; no voters:
I20260929 18:55:39.620357 11244 leader_election.cc:288] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:55:39.620424 11247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:39.620613 11244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:39.620658 11247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 23aca554ceda4118a6c26d81992d8ed7, State: Running, Role: LEADER
I20260929 18:55:39.620748 11247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aca554ceda4118a6c26d81992d8ed7" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41519 } }
I20260929 18:55:39.621198 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23aca554ceda4118a6c26d81992d8ed7. Latest consensus state: current_term: 1 leader_uuid: "23aca554ceda4118a6c26d81992d8ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aca554ceda4118a6c26d81992d8ed7" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41519 } } }
I20260929 18:55:39.621284 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:39.621335 11248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23aca554ceda4118a6c26d81992d8ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23aca554ceda4118a6c26d81992d8ed7" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41519 } } }
I20260929 18:55:39.621447 11248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:39.621502 11256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:39.621672 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11176
I20260929 18:55:39.621788 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:55:39.622016 29982 external_mini_cluster.cc:1428] Setting key cd5e391be505430be532ef4e00e21b61
I20260929 18:55:39.622014 11256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:55:39.623755 11256 catalog_manager.cc:1347] Generated new cluster ID: e21b656ee51248d5b96aa9b318057c7f
I20260929 18:55:39.623808 11256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:55:39 dist-test-slave-q2kh krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:39.632201 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:39.622480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.29.71.190:41519)
0929 18:55:39.622784 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:39.622798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:39.622822 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:55:39.622912 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:39.622918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:39.622959 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:39.623241 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:39.624074 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:39.625066 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:39.625831 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:39.626311 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:39.626444 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:39.628790 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:39.628794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:39.628795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:39.628806 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:39.630661 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:39.631029 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:39.631034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:39.631035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:39.631136 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:39.631351 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:39.631352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:39.631353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:39.631591 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:39.631755 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:39.631861 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:55:39.632003 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":62,"threads_started":1}
I20260929 18:55:39.633107 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:55:39.650130 11256 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:55:39.650894 11256 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:55:39.657968 11256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23aca554ceda4118a6c26d81992d8ed7: Generated new TSK 0
I20260929 18:55:39.658190 11256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:55:39.687026 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11176
2026-09-29T18:55:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1409 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:55:39 dist-test-slave-q2kh krb5kdc[11271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:55:39 dist-test-slave-q2kh krb5kdc[11271](info): set up 2 sockets
Sep 29 18:55:39 dist-test-slave-q2kh krb5kdc[11271](info): commencing operation
krb5kdc: starting...
W20260929 18:55:40.718348 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:55:40 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:55:40.837298 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33365
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:40.916144 11284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:40.916392 11284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:40.916431 11284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:40.917573 11284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:40.917641 11284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:40.917666 11284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:40.917694 11284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:40.917721 11284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:40.919179 11284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:40.919456 11284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:40.919677 11284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:40.924579 11290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:40.924612 11289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:40.924563 11292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:40.924736 11284 server_base.cc:1034] running on GCE node
I20260929 18:55:40.925042 11284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:40.925374 11284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:40.926622 11284 hybrid_clock.cc:648] HybridClock initialized: now 1790708140926607 us; error 68 us; skew 500 ppm
Sep 29 18:55:40 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708140, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:40.930398 11284 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:40.930959 11284 webserver.cc:460] Webserver started at http://127.29.71.190:40613/ using document root <none> and password file <none>
I20260929 18:55:40.931246 11284 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:40.931317 11284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:40.931461 11284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:40.932928 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "a47ecb1564b5485cb49b5a91477a3503"
format_stamp: "Formatted at 2026-09-29 18:55:40 on dist-test-slave-q2kh"
server_key: "cc7680f9d134798a6ee73a5cdbf0dd1f"
server_key_iv: "9b9ed31232f550acb5abae2f9a8cfa29"
server_key_version: "encryptionkey@0"
I20260929 18:55:40.933409 11284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "a47ecb1564b5485cb49b5a91477a3503"
format_stamp: "Formatted at 2026-09-29 18:55:40 on dist-test-slave-q2kh"
server_key: "cc7680f9d134798a6ee73a5cdbf0dd1f"
server_key_iv: "9b9ed31232f550acb5abae2f9a8cfa29"
server_key_version: "encryptionkey@0"
I20260929 18:55:40.935339 11284 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 18:55:40.936267 11299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:40.936614 11284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:55:40.936866 11284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "a47ecb1564b5485cb49b5a91477a3503"
format_stamp: "Formatted at 2026-09-29 18:55:40 on dist-test-slave-q2kh"
server_key: "cc7680f9d134798a6ee73a5cdbf0dd1f"
server_key_iv: "9b9ed31232f550acb5abae2f9a8cfa29"
server_key_version: "encryptionkey@0"
I20260929 18:55:40.936961 11284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:40.953541 11284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:40.956269 11284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:40.956446 11284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:40.961030 11284 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33365
I20260929 18:55:40.961088 11351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33365 every 8 connection(s)
I20260929 18:55:40.961455 11284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:40.961897 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11284
I20260929 18:55:40.962021 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:55:40.962281 29982 external_mini_cluster.cc:1428] Setting key e65caad3fb1e53a044cd1076f1daf735
I20260929 18:55:40.963114 11352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:40.965970 11352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Bootstrap starting.
I20260929 18:55:40.966717 11352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:40.967022 11352 log.cc:824] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:40.967816 11352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: No bootstrap required, opened a new log
I20260929 18:55:40.970464 11352 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:40.970621 11352 raft_consensus.cc:374] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 29 18:55:40 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:40.970664 11352 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47ecb1564b5485cb49b5a91477a3503, State: Initialized, Role: FOLLOWER
I20260929 18:55:40.970798 11352 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:40.970909 11352 raft_consensus.cc:388] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:55:40.970955 11352 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:55:40.971010 11352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:40.971771 11352 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:40.971877 11352 leader_election.cc:302] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47ecb1564b5485cb49b5a91477a3503; no voters:
I20260929 18:55:40.972064 11352 leader_election.cc:288] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:55:40.972131 11357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:40.972273 11357 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 LEADER]: Becoming Leader. State: Replica: a47ecb1564b5485cb49b5a91477a3503, State: Running, Role: LEADER
I20260929 18:55:40.972343 11352 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:40.972370 11357 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:40.972712 11357 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47ecb1564b5485cb49b5a91477a3503. Latest consensus state: current_term: 1 leader_uuid: "a47ecb1564b5485cb49b5a91477a3503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } } }
I20260929 18:55:40.972725 11358 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a47ecb1564b5485cb49b5a91477a3503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } } }
I20260929 18:55:40.972805 11357 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:40.972811 11358 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:40.973049 11364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:40.973608 11364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:55:40.976243 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:40.962844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.29.71.190:33365)
0929 18:55:40.963032 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:40.963043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:40.963075 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:55:40.963518 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:40.963524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:40.963569 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:40.963725 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:40.964576 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:40.965921 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:40.967230 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:40.967939 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:40.968053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:40.971281 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:40.971285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:40.971286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:40.971297 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:40.974574 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:40.975024 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:40.975027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:40.975028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:40.975164 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:40.975361 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:40.975362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:40.975363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:40.975687 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:40.975817 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:40.975932 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:55:40.976071 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260929 18:55:40.977449 11364 catalog_manager.cc:1347] Generated new cluster ID: 79ebc29266274cc1b540ed2ed30d9b8c
I20260929 18:55:40.977527 11364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:55:40.984871 11364 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:55:40.985366 11364 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:55:40.989017 11364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Generated new TSK 0
I20260929 18:55:40.989212 11364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:55:41.017486 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33365
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:41.099303 11378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:41.099499 11378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:41.099521 11378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:41.100651 11378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:41.100692 11378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:41.100716 11378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:55:41.102252 11378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:41.102497 11378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:41.102725 11378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:41.107704 11383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.107753 11386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.107730 11384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:41.108058 11378 server_base.cc:1034] running on GCE node
I20260929 18:55:41.108299 11378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:41.108548 11378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:41.109709 11378 hybrid_clock.cc:648] HybridClock initialized: now 1790708141109701 us; error 35 us; skew 500 ppm
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:41.112378 11378 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:55:41.112802 11378 webserver.cc:460] Webserver started at http://127.29.71.129:36467/ using document root <none> and password file <none>
I20260929 18:55:41.113044 11378 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:41.113112 11378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:41.113258 11378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:41.114377 11378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "26291254cd184d37860e029c77ba0e68"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "cf85e43d2915fdb1a3248c7b79d12ce6"
server_key_iv: "6341a23ae72795aab2604cc74f01f027"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.114727 11378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "26291254cd184d37860e029c77ba0e68"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "cf85e43d2915fdb1a3248c7b79d12ce6"
server_key_iv: "6341a23ae72795aab2604cc74f01f027"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.116060 11378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:41.116802 11393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.117010 11378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:41.117115 11378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "26291254cd184d37860e029c77ba0e68"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "cf85e43d2915fdb1a3248c7b79d12ce6"
server_key_iv: "6341a23ae72795aab2604cc74f01f027"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.117188 11378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:41.143191 11378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:41.145855 11378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:41.146073 11378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:41.146379 11378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:41.146747 11378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:41.146816 11378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.146864 11378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:41.146899 11378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.153911 11378 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:33045
I20260929 18:55:41.154006 11506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:33045 every 8 connection(s)
I20260929 18:55:41.154376 11378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:55:41.162506 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11378
I20260929 18:55:41.162621 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:55:41.162807 29982 external_mini_cluster.cc:1428] Setting key e5afce17033fd79b890ea65153fb06cc
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:41.168093 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.156712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:54853 (local address 127.29.71.190:33365)
0929 18:55:41.156764 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.156766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.157065 (+ 299us) server_negotiation.cc:407] Connection header received
0929 18:55:41.158056 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.158058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.158074 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.158138 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:41.159642 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.160350 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.161985 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.162247 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.162309 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.164768 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:41.164769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:41.164770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:41.164777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:41.166491 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.166978 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.166980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.166980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.167054 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.167355 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.167356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.167356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.167482 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:41.167567 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.167887 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.167989 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 18:55:41.168326 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.156702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:54853)
0929 18:55:41.156909 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.156960 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.157643 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.158269 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.158274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.159215 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:41.159539 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.159547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.160443 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.160444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.161903 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.161906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.162326 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.162327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.162440 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.163034 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:41.163042 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:41.164647 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:41.166619 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:41.166626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:41.166628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:41.166880 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:41.167131 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:41.167132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:41.167132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:41.167265 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:41.167557 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:41.167564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:41.167717 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.167860 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.168056 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260929 18:55:41.168921 11507 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33365
I20260929 18:55:41.169039 11507 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:41.169229 11507 heartbeater.cc:511] Master 127.29.71.190:33365 requested a full tablet report, sending...
I20260929 18:55:41.169895 11316 ts_manager.cc:194] Registered new tserver with Master: 26291254cd184d37860e029c77ba0e68 (127.29.71.129:33045)
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
I20260929 18:55:41.170899 11316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:54853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:55:41.188868 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33365
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:41.271242 11512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:41.271479 11512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:41.271517 11512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:41.273037 11512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:41.273116 11512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:41.273157 11512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:55:41.274761 11512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:41.275015 11512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:41.275271 11512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:41.280432 11517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.280440 11520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:41.280625 11512 server_base.cc:1034] running on GCE node
W20260929 18:55:41.280491 11518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:41.280896 11512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:41.281157 11512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:41.282322 11512 hybrid_clock.cc:648] HybridClock initialized: now 1790708141282320 us; error 38 us; skew 500 ppm
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:41.284920 11512 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:55:41.285423 11512 webserver.cc:460] Webserver started at http://127.29.71.130:37937/ using document root <none> and password file <none>
I20260929 18:55:41.285663 11512 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:41.285732 11512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:41.285866 11512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:41.286990 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "eeaa393ff7c34a7fafcded1577613cad"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "3275586fedce058469f0f2a4d1d23500"
server_key_iv: "b5375e5a43750a5edce3e3f62ec09d67"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.287353 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "eeaa393ff7c34a7fafcded1577613cad"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "3275586fedce058469f0f2a4d1d23500"
server_key_iv: "b5375e5a43750a5edce3e3f62ec09d67"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.288632 11512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:55:41.289343 11527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.289515 11512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:55:41.289609 11512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "eeaa393ff7c34a7fafcded1577613cad"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "3275586fedce058469f0f2a4d1d23500"
server_key_iv: "b5375e5a43750a5edce3e3f62ec09d67"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.289692 11512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:41.297191 11512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:41.298959 11512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:41.299124 11512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:41.299348 11512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:41.299650 11512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:41.299702 11512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.299742 11512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:41.299770 11512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.305833 11512 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:44241
I20260929 18:55:41.305873 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:44241 every 8 connection(s)
I20260929 18:55:41.306213 11512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:41.313707 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11512
I20260929 18:55:41.313822 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:55:41.314008 29982 external_mini_cluster.cc:1428] Setting key 185f7245c7e42fae43dad88efbf81f2a
I20260929 18:55:41.315591 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.306927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:41335 (local address 127.29.71.190:33365)
0929 18:55:41.306977 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.306979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.307160 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:55:41.307625 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.307627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.307637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.307704 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:41.308625 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.309140 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.310212 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.310459 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.310527 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.312604 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:41.312607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:41.312607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:41.312615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:41.314014 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.314542 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.314543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.314544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.314602 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.314938 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.314940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.314940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.315064 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:41.315143 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.315419 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.315500 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:55:41.315796 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.306869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:41335)
0929 18:55:41.307026 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.307067 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.307535 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.307755 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.307759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.308315 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:41.308513 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.308518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.309223 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.309224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.310047 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.310049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.310540 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.310541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.310639 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.311161 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:41.311169 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:41.312501 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:41.314108 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:41.314114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:41.314116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:41.314449 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:41.314732 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:41.314733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:41.314733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:41.314877 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:41.315134 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:41.315140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:41.315281 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.315398 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.315559 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260929 18:55:41.316303 11641 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33365
I20260929 18:55:41.316416 11641 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:41.316623 11641 heartbeater.cc:511] Master 127.29.71.190:33365 requested a full tablet report, sending...
I20260929 18:55:41.317044 11316 ts_manager.cc:194] Registered new tserver with Master: eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
I20260929 18:55:41.317433 11316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:41335
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:55:41.343211 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33365
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:41.426880 11646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:41.427115 11646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:41.427143 11646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:41.428485 11646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:41.428534 11646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:41.428565 11646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:55:41.429960 11646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:41.430147 11646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:41.430423 11646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:41.435568 11651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.435598 11652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.435618 11654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:41.435846 11646 server_base.cc:1034] running on GCE node
I20260929 18:55:41.436111 11646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:41.436376 11646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:41.437559 11646 hybrid_clock.cc:648] HybridClock initialized: now 1790708141437551 us; error 35 us; skew 500 ppm
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:41.440028 11646 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:55:41.440481 11646 webserver.cc:460] Webserver started at http://127.29.71.131:38959/ using document root <none> and password file <none>
I20260929 18:55:41.440681 11646 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:41.440721 11646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:41.440806 11646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:41.441907 11646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "761eefa69d9340b8ad2ccfb7a61b6000"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "337b9bb5b33d6628f526b5bcf976d4f7"
server_key_iv: "888ba4b9612a9a2ff4a1764323fbb1e8"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.442322 11646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "761eefa69d9340b8ad2ccfb7a61b6000"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "337b9bb5b33d6628f526b5bcf976d4f7"
server_key_iv: "888ba4b9612a9a2ff4a1764323fbb1e8"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.443571 11646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:55:41.444286 11661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.444442 11646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:55:41.444525 11646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "761eefa69d9340b8ad2ccfb7a61b6000"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "337b9bb5b33d6628f526b5bcf976d4f7"
server_key_iv: "888ba4b9612a9a2ff4a1764323fbb1e8"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.444576 11646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:41.460589 11646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:41.462970 11646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:41.463089 11646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:41.463286 11646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:41.463640 11646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:41.463675 11646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.463699 11646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:41.463749 11646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.469498 11646 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:37295
I20260929 18:55:41.469606 11774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:37295 every 8 connection(s)
I20260929 18:55:41.469897 11646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:41.478359 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11646
I20260929 18:55:41.478434 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:55:41.478613 29982 external_mini_cluster.cc:1428] Setting key 1951b19f99174c02df0c9f96d35cfedd
I20260929 18:55:41.479682 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.470680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:53655 (local address 127.29.71.190:33365)
0929 18:55:41.470738 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.470740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.470941 (+ 201us) server_negotiation.cc:407] Connection header received
0929 18:55:41.471440 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.471441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.471451 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.471511 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:41.472531 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.473232 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.474364 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.474564 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.474624 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.476768 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:41.476770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:41.476770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:41.476777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:41.478060 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.478574 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.478576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.478576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.478640 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.478940 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.478941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.478942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.479097 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:41.479172 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.479501 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.479586 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:55:41.479943 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.470654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:53655)
0929 18:55:41.470803 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.470844 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.471335 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.471565 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.471568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.472234 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:41.472437 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.472442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.473327 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.473328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.474269 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.474273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.474659 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.474661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.474769 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.475265 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:41.475273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:41.476688 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:41.478147 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:41.478152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:41.478191 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:41.478448 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:41.478711 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:41.478712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:41.478713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:41.478872 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:41.479169 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:41.479177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:41.479347 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.479457 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.479643 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260929 18:55:41.480353 11775 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33365
I20260929 18:55:41.480458 11775 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:41.480659 11775 heartbeater.cc:511] Master 127.29.71.190:33365 requested a full tablet report, sending...
I20260929 18:55:41.481019 11316 ts_manager.cc:194] Registered new tserver with Master: 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295)
I20260929 18:55:41.481400 11316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:53655
I20260929 18:55:41.490581 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:55:41.498458 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.491894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.29.71.190:33365)
0929 18:55:41.491954 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.491956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.492078 (+ 122us) server_negotiation.cc:407] Connection header received
0929 18:55:41.492123 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.492124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.492134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.492180 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:41.492807 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.493306 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.494072 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.494303 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.494356 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.495523 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:41.495524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:41.495525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:41.495531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:41.497192 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.497542 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.497543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.497543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.497604 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:41.497875 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:41.497876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:41.497876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:41.498062 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:41.498210 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.498325 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.498391 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:55:41.500104 11316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:55:41.501772 11316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:55:41.508808 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.504296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33045 (local address 127.0.0.1:46956)
0929 18:55:41.504482 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.504523 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.504617 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.505143 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.505144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.505147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:41.505209 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.505210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.506614 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.506615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.507653 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.507655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.508482 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.508558 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.508572 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.508652 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.508706 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":32,"thread_start_us":72,"threads_started":1}
I20260929 18:55:41.508807 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.504415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.29.71.129:33045)
0929 18:55:41.504715 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.504720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.504960 (+ 240us) server_negotiation.cc:407] Connection header received
0929 18:55:41.505003 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.505005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.505020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.505094 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:41.505370 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.506481 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.507765 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.508391 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508465 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.508518 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.508677 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.508719 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":98,"threads_started":1}
I20260929 18:55:41.509225 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.504296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:37295 (local address 127.0.0.1:59816)
0929 18:55:41.504722 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.504731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.504773 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.505122 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.505125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.505127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:41.505218 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.505219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.506587 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.506589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.508000 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.508003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.508985 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.509049 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.509053 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.509110 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.509155 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":106,"thread_start_us":71,"threads_started":1}
I20260929 18:55:41.509264 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.504415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.29.71.131:37295)
0929 18:55:41.504634 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.504639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.504838 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:55:41.504935 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.504938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.504956 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.505030 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:41.505301 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.506480 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508099 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.508829 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508899 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.508951 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.509140 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.509184 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260929 18:55:41.509733 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.504517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:44241 (local address 127.0.0.1:52360)
0929 18:55:41.504834 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.504843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.504886 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.505138 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.505138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.505139 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:41.505216 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.505218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.506958 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.506959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.508034 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.508036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.508846 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.509560 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.509564 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.509631 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.509669 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":119,"threads_started":1}
I20260929 18:55:41.509769 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.504673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.29.71.130:44241)
0929 18:55:41.504941 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.504945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.504955 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:55:41.504997 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.504999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.505016 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.505094 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:41.505934 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.506812 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508119 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.508704 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.508762 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.508812 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.509650 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.509687 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":76,"threads_started":1}
I20260929 18:55:41.509639 11441 tablet_service.cc:1467] Processing CreateTablet for tablet 79a2971606ba474796d0f691535150d3 (DEFAULT_TABLE table=test-table [id=7b8d63e815c845689fc608a62577e785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:41.510003 11441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a2971606ba474796d0f691535150d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:41.509999 11709 tablet_service.cc:1467] Processing CreateTablet for tablet 79a2971606ba474796d0f691535150d3 (DEFAULT_TABLE table=test-table [id=7b8d63e815c845689fc608a62577e785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:41.510349 11709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a2971606ba474796d0f691535150d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:41.510717 11575 tablet_service.cc:1467] Processing CreateTablet for tablet 79a2971606ba474796d0f691535150d3 (DEFAULT_TABLE table=test-table [id=7b8d63e815c845689fc608a62577e785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:41.510958 11575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79a2971606ba474796d0f691535150d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:41.513540 11793 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Bootstrap starting.
I20260929 18:55:41.513541 11792 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Bootstrap starting.
I20260929 18:55:41.514093 11792 tablet_bootstrap.cc:654] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:41.514106 11793 tablet_bootstrap.cc:654] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:41.514328 11794 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Bootstrap starting.
I20260929 18:55:41.514711 11792 log.cc:824] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:41.514889 11794 tablet_bootstrap.cc:654] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:41.515480 11793 log.cc:824] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:41.515480 11794 log.cc:824] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:41.515626 11792 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: No bootstrap required, opened a new log
I20260929 18:55:41.515702 11792 ts_tablet_manager.cc:1397] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:55:41.516235 11794 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: No bootstrap required, opened a new log
I20260929 18:55:41.516299 11793 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: No bootstrap required, opened a new log
I20260929 18:55:41.516322 11794 ts_tablet_manager.cc:1397] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:55:41.516366 11793 ts_tablet_manager.cc:1397] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:55:41.517789 11792 raft_consensus.cc:348] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.517987 11792 raft_consensus.cc:374] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:41.517874 11794 raft_consensus.cc:348] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.518028 11792 raft_consensus.cc:729] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeaa393ff7c34a7fafcded1577613cad, State: Initialized, Role: FOLLOWER
I20260929 18:55:41.518036 11794 raft_consensus.cc:374] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:41.518071 11794 raft_consensus.cc:729] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26291254cd184d37860e029c77ba0e68, State: Initialized, Role: FOLLOWER
I20260929 18:55:41.518211 11792 consensus_queue.cc:260] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.518250 11794 consensus_queue.cc:260] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.518358 11793 raft_consensus.cc:348] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.518504 11793 raft_consensus.cc:374] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:41.518522 11792 ts_tablet_manager.cc:1428] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:55:41.518541 11793 raft_consensus.cc:729] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761eefa69d9340b8ad2ccfb7a61b6000, State: Initialized, Role: FOLLOWER
I20260929 18:55:41.518539 11794 ts_tablet_manager.cc:1428] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:55:41.518635 11641 heartbeater.cc:503] Master 127.29.71.190:33365 was elected leader, sending a full tablet report...
I20260929 18:55:41.518658 11507 heartbeater.cc:503] Master 127.29.71.190:33365 was elected leader, sending a full tablet report...
I20260929 18:55:41.518646 11793 consensus_queue.cc:260] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.518903 11793 ts_tablet_manager.cc:1428] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:55:41.519032 11775 heartbeater.cc:503] Master 127.29.71.190:33365 was elected leader, sending a full tablet report...
W20260929 18:55:41.556774 11642 tablet.cc:2369] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:55:41.655185 11508 tablet.cc:2369] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:55:41.665187 11799 raft_consensus.cc:482] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:41.665354 11799 raft_consensus.cc:504] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.665771 11799 leader_election.cc:288] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295), eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
I20260929 18:55:41.668854 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.666051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:37295 (local address 127.29.71.129:36557)
0929 18:55:41.666120 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.666133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.666231 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.666416 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.666417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.666423 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:41.666515 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.666518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.667230 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.667231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.668026 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.668028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.668583 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.668584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.668668 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.668671 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.668723 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.668753 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":13}
I20260929 18:55:41.668864 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.666201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:36557 (local address 127.29.71.131:37295)
0929 18:55:41.666256 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.666259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.666271 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:55:41.666301 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.666302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.666311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.666405 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:41.666607 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.667147 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.668120 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.668499 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.668555 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.668562 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.668746 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.668793 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:55:41.668927 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.666292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:41239 (local address 127.29.71.130:44241)
0929 18:55:41.666359 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.666361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.666460 (+ 99us) server_negotiation.cc:407] Connection header received
0929 18:55:41.666514 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.666515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.666528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.666582 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:41.666744 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.667218 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.668155 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.668562 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.668634 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.668643 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.668806 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.668856 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:55:41.668980 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.666234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:44241 (local address 127.29.71.129:41239)
0929 18:55:41.666390 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.666399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.666454 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.666606 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.666607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.666608 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:41.666676 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.666677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.667299 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.667300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.668035 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.668037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.668650 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.668651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.668720 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.668725 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:55:41.668783 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:55:41.668819 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260929 18:55:41.669212 11729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a2971606ba474796d0f691535150d3" candidate_uuid: "26291254cd184d37860e029c77ba0e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" is_pre_election: true
I20260929 18:55:41.669344 11729 raft_consensus.cc:2454] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26291254cd184d37860e029c77ba0e68 in term 0.
I20260929 18:55:41.669613 11394 leader_election.cc:302] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26291254cd184d37860e029c77ba0e68, 761eefa69d9340b8ad2ccfb7a61b6000; no voters:
I20260929 18:55:41.669659 11595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a2971606ba474796d0f691535150d3" candidate_uuid: "26291254cd184d37860e029c77ba0e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeaa393ff7c34a7fafcded1577613cad" is_pre_election: true
I20260929 18:55:41.669775 11799 raft_consensus.cc:2781] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:55:41.669845 11799 raft_consensus.cc:482] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:55:41.669806 11595 raft_consensus.cc:2454] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26291254cd184d37860e029c77ba0e68 in term 0.
I20260929 18:55:41.669878 11799 raft_consensus.cc:3037] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:41.670708 11799 raft_consensus.cc:504] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.670840 11799 leader_election.cc:288] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 1 election: Requested vote from peers 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295), eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
I20260929 18:55:41.671028 11595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a2971606ba474796d0f691535150d3" candidate_uuid: "26291254cd184d37860e029c77ba0e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeaa393ff7c34a7fafcded1577613cad"
I20260929 18:55:41.671038 11729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79a2971606ba474796d0f691535150d3" candidate_uuid: "26291254cd184d37860e029c77ba0e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761eefa69d9340b8ad2ccfb7a61b6000"
I20260929 18:55:41.671119 11595 raft_consensus.cc:3037] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:41.671118 11729 raft_consensus.cc:3037] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:41.671835 11595 raft_consensus.cc:2454] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26291254cd184d37860e029c77ba0e68 in term 1.
I20260929 18:55:41.671878 11729 raft_consensus.cc:2454] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26291254cd184d37860e029c77ba0e68 in term 1.
I20260929 18:55:41.672027 11394 leader_election.cc:302] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26291254cd184d37860e029c77ba0e68, 761eefa69d9340b8ad2ccfb7a61b6000; no voters:
I20260929 18:55:41.672206 11799 raft_consensus.cc:2781] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:41.672469 11799 raft_consensus.cc:686] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 1 LEADER]: Becoming Leader. State: Replica: 26291254cd184d37860e029c77ba0e68, State: Running, Role: LEADER
I20260929 18:55:41.672608 11799 consensus_queue.cc:237] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.673549 11316 catalog_manager.cc:5495] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26291254cd184d37860e029c77ba0e68 (127.29.71.129). New cstate: current_term: 1 leader_uuid: "26291254cd184d37860e029c77ba0e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } health_report { overall_health: UNKNOWN } } }
I20260929 18:55:41.690366 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.687813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.29.71.129:33045)
0929 18:55:41.687872 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.687874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.687884 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:55:41.687931 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.687932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.687951 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.688016 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:55:41.688127 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.688589 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.689296 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.689496 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.689556 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.689749 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:55:41.689914 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:55:41.690024 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:41.690089 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:55:41.690180 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:55:41.692303 11595 raft_consensus.cc:1261] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Refusing update from remote peer 26291254cd184d37860e029c77ba0e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:41.692303 11729 raft_consensus.cc:1261] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 FOLLOWER]: Refusing update from remote peer 26291254cd184d37860e029c77ba0e68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:41.692723 11802 consensus_queue.cc:1035] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:41.692919 11799 consensus_queue.cc:1035] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:41.695660 11806 mvcc.cc:205] Tried to move back new op lower bound from 7334740548368723968 to 7334740548291764224. Current Snapshot: MvccSnapshot[applied={T|T < 7334740548368723968}]
I20260929 18:55:41.695994 11807 mvcc.cc:205] Tried to move back new op lower bound from 7334740548368723968 to 7334740548291764224. Current Snapshot: MvccSnapshot[applied={T|T < 7334740548368723968}]
I20260929 18:55:41.697419 11314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:55:41.697525 11314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:55:41.698272 11314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Sending DeleteTablet for 3 replicas of tablet 79a2971606ba474796d0f691535150d3
I20260929 18:55:41.698647 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11378
I20260929 18:55:41.698616 11441 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a2971606ba474796d0f691535150d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:41 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46956
I20260929 18:55:41.698662 11575 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a2971606ba474796d0f691535150d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:41 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:52360
I20260929 18:55:41.698891 11709 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a2971606ba474796d0f691535150d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:41 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59816
I20260929 18:55:41.699559 11815 tablet_replica.cc:330] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: stopping tablet replica
I20260929 18:55:41.699676 11815 raft_consensus.cc:2229] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:55:41.699788 11815 raft_consensus.cc:2258] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:41.700241 11815 ts_tablet_manager.cc:1905] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:55:41.702325 11816 tablet_replica.cc:330] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: stopping tablet replica
I20260929 18:55:41.702349 11815 ts_tablet_manager.cc:1918] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:55:41.702417 11815 log.cc:1196] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/79a2971606ba474796d0f691535150d3
I20260929 18:55:41.702463 11816 raft_consensus.cc:2229] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:55:41.702577 11816 raft_consensus.cc:2258] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:41.702687 11815 ts_tablet_manager.cc:1939] T 79a2971606ba474796d0f691535150d3 P eeaa393ff7c34a7fafcded1577613cad: Deleting consensus metadata
I20260929 18:55:41.703120 11816 ts_tablet_manager.cc:1905] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:55:41.703406 11300 catalog_manager.cc:4841] TS eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241): tablet 79a2971606ba474796d0f691535150d3 (table test-table [id=7b8d63e815c845689fc608a62577e785]) successfully deleted
I20260929 18:55:41.704983 11816 ts_tablet_manager.cc:1918] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:55:41.705052 11816 log.cc:1196] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/79a2971606ba474796d0f691535150d3
I20260929 18:55:41.705360 11816 ts_tablet_manager.cc:1939] T 79a2971606ba474796d0f691535150d3 P 761eefa69d9340b8ad2ccfb7a61b6000: Deleting consensus metadata
I20260929 18:55:41.705878 11300 catalog_manager.cc:4841] TS 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295): tablet 79a2971606ba474796d0f691535150d3 (table test-table [id=7b8d63e815c845689fc608a62577e785]) successfully deleted
W20260929 18:55:41.705879 11302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:33045 (error 108)
I20260929 18:55:41.706198 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11512
I20260929 18:55:41.707006 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.706834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33045 (local address 127.0.0.1:46976)
0929 18:55:41.706904 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.706956 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:55:41.707104 11302 catalog_manager.cc:4584] TS 26291254cd184d37860e029c77ba0e68 (127.29.71.129:33045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79a2971606ba474796d0f691535150d3: Network error: Client connection negotiation failed: client connection to 127.29.71.129:33045: connect: Connection refused (error 111)
I20260929 18:55:41.712728 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11646
I20260929 18:55:41.717437 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11284
I20260929 18:55:41.722673 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33365
--webserver_interface=127.29.71.190
--webserver_port=40613
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:41.807765 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:41.807963 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:41.807986 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:41.809062 11818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:41.809098 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:41.809111 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:41.809124 11818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:41.809136 11818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:41.810613 11818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:41.810809 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:41.810969 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:41.819422 11827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.819473 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:41.819651 11818 server_base.cc:1034] running on GCE node
W20260929 18:55:41.819429 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:41.820039 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:41.820374 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:41.821571 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790708141821568 us; error 51 us; skew 500 ppm
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:41.828796 11818 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:41.829322 11818 webserver.cc:460] Webserver started at http://127.29.71.190:40613/ using document root <none> and password file <none>
I20260929 18:55:41.829525 11818 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:41.829576 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:41.830972 11818 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:41.831633 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.831911 11818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:55:41.832003 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "a47ecb1564b5485cb49b5a91477a3503"
format_stamp: "Formatted at 2026-09-29 18:55:40 on dist-test-slave-q2kh"
server_key: "cc7680f9d134798a6ee73a5cdbf0dd1f"
server_key_iv: "9b9ed31232f550acb5abae2f9a8cfa29"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.832353 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:41.850041 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:41.852205 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:41.852313 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:41.856271 11818 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33365
I20260929 18:55:41.856320 11886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33365 every 8 connection(s)
I20260929 18:55:41.856707 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:41.857942 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11818
I20260929 18:55:41.858290 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:33045
--local_ip_for_outbound_sockets=127.29.71.129
--tserver_master_addrs=127.29.71.190:33365
--webserver_port=36467
--webserver_interface=127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:41.858505 11887 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:55:41.859192 11887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Bootstrap starting.
I20260929 18:55:41.867566 11887 log.cc:824] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:41.869611 11887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:55:41.869944 11887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Bootstrap complete.
I20260929 18:55:41.872430 11887 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:41.872661 11887 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47ecb1564b5485cb49b5a91477a3503, State: Initialized, Role: FOLLOWER
I20260929 18:55:41.872830 11887 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:41.872906 11887 raft_consensus.cc:388] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:55:41.872953 11887 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:55:41.872995 11887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:55:41.874022 11887 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:41.874109 11887 leader_election.cc:302] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47ecb1564b5485cb49b5a91477a3503; no voters:
I20260929 18:55:41.874343 11887 leader_election.cc:288] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:55:41.874387 11891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:55:41.874505 11891 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 LEADER]: Becoming Leader. State: Replica: a47ecb1564b5485cb49b5a91477a3503, State: Running, Role: LEADER
I20260929 18:55:41.874620 11887 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:41.874599 11891 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:41.874929 11891 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47ecb1564b5485cb49b5a91477a3503. Latest consensus state: current_term: 2 leader_uuid: "a47ecb1564b5485cb49b5a91477a3503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } } }
I20260929 18:55:41.875010 11891 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:41.874933 11893 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a47ecb1564b5485cb49b5a91477a3503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } } }
I20260929 18:55:41.875164 11893 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:41.875222 11896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:41.876063 11896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:55:41.876170 11896 catalog_manager.cc:1259] Loaded cluster ID: 79ebc29266274cc1b540ed2ed30d9b8c
I20260929 18:55:41.876206 11896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:55:41.876459 11896 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 18:55:41.881110 11896 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 18:55:41.883841 11896 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:55:41.884047 11896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Loaded TSK: 0
I20260929 18:55:41.884267 11896 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:55:41.941277 11889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:41.941519 11889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:41.941560 11889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:41.942793 11889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:41.942859 11889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:41.942903 11889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:55:41.944373 11889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:33045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=36467
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:41.944635 11889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:41.944855 11889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:41.949718 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.949740 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:41.949760 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:41.950047 11889 server_base.cc:1034] running on GCE node
I20260929 18:55:41.950261 11889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:41.950511 11889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:41.951680 11889 hybrid_clock.cc:648] HybridClock initialized: now 1790708141951667 us; error 30 us; skew 500 ppm
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:41.954325 11889 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:55:41.954809 11889 webserver.cc:460] Webserver started at http://127.29.71.129:36467/ using document root <none> and password file <none>
I20260929 18:55:41.955041 11889 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:41.955120 11889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:41.956465 11889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:41.957152 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.957304 11889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:41.957391 11889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "26291254cd184d37860e029c77ba0e68"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "cf85e43d2915fdb1a3248c7b79d12ce6"
server_key_iv: "6341a23ae72795aab2604cc74f01f027"
server_key_version: "encryptionkey@0"
I20260929 18:55:41.957705 11889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:41.966862 11889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:41.968879 11889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:41.968987 11889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:41.969187 11889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:41.969655 11931 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:55:41.971134 11889 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:55:41.971189 11889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 18:55:41.971215 11889 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:55:41.971801 11889 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:55:41.971843 11889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 18:55:41.971897 11931 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Bootstrap starting.
I20260929 18:55:41.981655 11889 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:33045
I20260929 18:55:41.982194 11889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:55:41.982385 12038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:33045 every 8 connection(s)
I20260929 18:55:41.982995 11931 log.cc:824] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:41.983341 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 11889
I20260929 18:55:41.983629 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:44241
--local_ip_for_outbound_sockets=127.29.71.130
--tserver_master_addrs=127.29.71.190:33365
--webserver_port=37937
--webserver_interface=127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:41.985075 11931 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:55:41.985379 11931 tablet_bootstrap.cc:492] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Bootstrap complete.
I20260929 18:55:41.986527 11931 ts_tablet_manager.cc:1397] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260929 18:55:41.988543 11931 raft_consensus.cc:348] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.988885 11931 raft_consensus.cc:729] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26291254cd184d37860e029c77ba0e68, State: Initialized, Role: FOLLOWER
I20260929 18:55:41.989022 11931 consensus_queue.cc:260] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } }
I20260929 18:55:41.989300 11931 ts_tablet_manager.cc:1428] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
Sep 29 18:55:41 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708141, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:42.002379 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.987033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:43319 (local address 127.29.71.190:33365)
0929 18:55:41.990356 (+ 3323us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:41.990366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:41.990406 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:55:41.990536 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:41.990542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:41.990599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:41.990767 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:41.991786 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.993348 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.994786 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.995476 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.995600 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.998218 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:41.998223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:41.998223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:41.998234 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:42.000312 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.000865 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.000869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.000869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.000982 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.001270 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.001271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.001271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.001574 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:42.001678 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.002014 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.002152 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3219,"thread_start_us":75,"threads_started":1}
I20260929 18:55:42.002380 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:41.986718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:43319)
0929 18:55:41.986911 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:41.986959 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:41.988010 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:41.990829 (+ 2819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:41.990842 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:41.991289 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:41.991586 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.991594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.993460 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.993461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.994692 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:41.994695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:41.995578 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:41.995579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:41.995719 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:41.996368 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:41.996379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:41.998078 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:42.000411 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:42.000420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:42.000423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:42.000768 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:42.001058 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:42.001059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:42.001059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:42.001193 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:42.001660 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:42.001667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:42.001819 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.001935 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.002102 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260929 18:55:42.003726 12039 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33365
I20260929 18:55:42.003854 12039 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:42.004081 12039 heartbeater.cc:511] Master 127.29.71.190:33365 requested a full tablet report, sending...
I20260929 18:55:42.004884 11851 ts_manager.cc:194] Registered new tserver with Master: 26291254cd184d37860e029c77ba0e68 (127.29.71.129:33045)
I20260929 18:55:42.006098 11851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:43319
I20260929 18:55:42.007085 12039 heartbeater.cc:503] Master 127.29.71.190:33365 was elected leader, sending a full tablet report...
I20260929 18:55:42.014235 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.005774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.29.71.129:33045)
0929 18:55:42.010525 (+ 4751us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.010529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.010538 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:55:42.010570 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.010572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.010587 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.010814 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:42.010861 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.011759 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.013147 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.013769 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.013879 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.013936 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.014093 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.014134 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4720,"thread_start_us":66,"threads_started":1}
I20260929 18:55:42.014235 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.005524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33045 (local address 127.0.0.1:46992)
0929 18:55:42.010293 (+ 4769us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.010333 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.010399 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.010676 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:42.010678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:42.010681 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:42.010747 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.010749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.011860 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.011862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.013042 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.013046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.013881 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.013882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.013964 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.013984 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.014075 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.014130 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4727,"thread_start_us":100,"threads_started":1}
I20260929 18:55:42.014797 11965 tablet_service.cc:1514] Processing DeleteTablet for tablet 79a2971606ba474796d0f691535150d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:41 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46992
I20260929 18:55:42.015419 12048 tablet_replica.cc:330] stopping tablet replica
I20260929 18:55:42.015548 12048 raft_consensus.cc:2229] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:55:42.015633 12048 raft_consensus.cc:2258] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:42.016036 12048 ts_tablet_manager.cc:1905] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:55:42.018355 12048 ts_tablet_manager.cc:1918] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:55:42.018456 12048 log.cc:1196] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/79a2971606ba474796d0f691535150d3
I20260929 18:55:42.018690 12048 ts_tablet_manager.cc:1939] T 79a2971606ba474796d0f691535150d3 P 26291254cd184d37860e029c77ba0e68: Deleting consensus metadata
I20260929 18:55:42.019243 11837 catalog_manager.cc:4841] TS 26291254cd184d37860e029c77ba0e68 (127.29.71.129:33045): tablet 79a2971606ba474796d0f691535150d3 (table test-table [id=7b8d63e815c845689fc608a62577e785]) successfully deleted
W20260929 18:55:42.071794 12042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:42.071997 12042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:42.072021 12042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:42.073076 12042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:42.073109 12042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:42.073135 12042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:55:42.074560 12042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:44241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=37937
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:42.074765 12042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:42.075055 12042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:42.080145 12053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.080188 12054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.080134 12056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.080394 12042 server_base.cc:1034] running on GCE node
I20260929 18:55:42.080617 12042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:42.080873 12042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:42.082048 12042 hybrid_clock.cc:648] HybridClock initialized: now 1790708142082042 us; error 39 us; skew 500 ppm
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:42.084684 12042 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:55:42.085206 12042 webserver.cc:460] Webserver started at http://127.29.71.130:37937/ using document root <none> and password file <none>
I20260929 18:55:42.085454 12042 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:42.085536 12042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:42.087008 12042 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.087626 12063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.087796 12042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:55:42.087896 12042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "eeaa393ff7c34a7fafcded1577613cad"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "3275586fedce058469f0f2a4d1d23500"
server_key_iv: "b5375e5a43750a5edce3e3f62ec09d67"
server_key_version: "encryptionkey@0"
I20260929 18:55:42.088244 12042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:42.106158 12042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:42.108246 12042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:42.108420 12042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:42.108669 12042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:42.108994 12042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:42.109046 12042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.109081 12042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:42.109108 12042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.115145 12042 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:44241
I20260929 18:55:42.115211 12176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:44241 every 8 connection(s)
I20260929 18:55:42.115581 12042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:55:42.120807 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 12042
I20260929 18:55:42.121134 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:37295
--local_ip_for_outbound_sockets=127.29.71.131
--tserver_master_addrs=127.29.71.190:33365
--webserver_port=38959
--webserver_interface=127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:42.125792 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.116433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:53635 (local address 127.29.71.190:33365)
0929 18:55:42.116523 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.116526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.116787 (+ 261us) server_negotiation.cc:407] Connection header received
0929 18:55:42.117328 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.117329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.117347 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.117405 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:42.118538 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.119047 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.120046 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.120242 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.120302 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.122891 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:42.122893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:42.122894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:42.122901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:42.124344 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.124785 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.124786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.124787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.124846 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.125110 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.125111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.125112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.125231 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:42.125321 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.125606 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.125700 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:55:42.125986 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.116424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:53635)
0929 18:55:42.116651 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.116695 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.117218 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.117461 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:42.117465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:42.118148 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:42.118445 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.118451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.119148 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.119149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.119962 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.119964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.120345 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.120346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.120455 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.121062 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:42.121072 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:42.122787 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:42.124467 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:42.124474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:42.124476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:42.124702 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:42.124924 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:42.124925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:42.124925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:42.125038 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:42.125312 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:42.125318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:42.125456 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.125563 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.125723 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":116,"threads_started":1}
I20260929 18:55:42.126472 12177 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33365
I20260929 18:55:42.126569 12177 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:42.126768 12177 heartbeater.cc:511] Master 127.29.71.190:33365 requested a full tablet report, sending...
I20260929 18:55:42.127167 11851 ts_manager.cc:194] Registered new tserver with Master: eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
I20260929 18:55:42.127542 11851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:53635
W20260929 18:55:42.202615 12180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:42.202863 12180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:42.202903 12180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:42.204095 12180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:42.204162 12180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:42.204209 12180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:55:42.205678 12180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:37295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=38959
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:42.205956 12180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:42.206233 12180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:42.210994 12188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.210994 12186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.211124 12185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.211479 12180 server_base.cc:1034] running on GCE node
I20260929 18:55:42.211704 12180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:42.211978 12180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:42.213155 12180 hybrid_clock.cc:648] HybridClock initialized: now 1790708142213146 us; error 50 us; skew 500 ppm
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:42.215785 12180 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:55:42.216264 12180 webserver.cc:460] Webserver started at http://127.29.71.131:38959/ using document root <none> and password file <none>
I20260929 18:55:42.216498 12180 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:42.216591 12180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:42.217914 12180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.218552 12195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.218708 12180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.218808 12180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "761eefa69d9340b8ad2ccfb7a61b6000"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "337b9bb5b33d6628f526b5bcf976d4f7"
server_key_iv: "888ba4b9612a9a2ff4a1764323fbb1e8"
server_key_version: "encryptionkey@0"
I20260929 18:55:42.219172 12180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:42.249823 12180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:42.252297 12180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:42.252476 12180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:42.252727 12180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:42.253053 12180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:42.253109 12180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.253149 12180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:42.253177 12180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.259016 12180 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:37295
I20260929 18:55:42.259056 12308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:37295 every 8 connection(s)
I20260929 18:55:42.259400 12180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:42.266541 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 12180
I20260929 18:55:42.268959 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.260187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:49903 (local address 127.29.71.190:33365)
0929 18:55:42.260264 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.260266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.260489 (+ 223us) server_negotiation.cc:407] Connection header received
0929 18:55:42.261041 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.261042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.261052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.261108 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:42.262051 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.262550 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.263488 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.263673 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.263728 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.265727 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:42.265728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:42.265729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:42.265736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:42.267640 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.268041 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.268042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.268042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.268099 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.268334 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.268335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.268335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.268444 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:42.268533 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.268791 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.268870 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:55:42.269160 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.260199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:49903)
0929 18:55:42.260366 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.260401 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.260935 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.261182 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:42.261185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:42.261757 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:42.261951 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.261955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.262622 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.262622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.263409 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.263411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.263746 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.263746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.263841 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.264318 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:42.264326 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:42.265619 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:42.267722 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:42.267729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:42.267731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:42.267944 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:42.268165 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:42.268165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:42.268166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:42.268266 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:42.268525 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:42.268531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:42.268662 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.268771 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.268923 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260929 18:55:42.269632 12309 heartbeater.cc:347] Connected to a master server at 127.29.71.190:33365
I20260929 18:55:42.269745 12309 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:42.269934 12309 heartbeater.cc:511] Master 127.29.71.190:33365 requested a full tablet report, sending...
I20260929 18:55:42.270340 11851 ts_manager.cc:194] Registered new tserver with Master: 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295)
I20260929 18:55:42.270707 11851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:49903
I20260929 18:55:42.274428 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.266999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.29.71.190:33365)
0929 18:55:42.267134 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.267136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.267207 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:55:42.267270 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.267271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.267282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.267347 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:42.267981 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.268523 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.269746 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.270096 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.270207 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.271626 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:42.271627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:42.271628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:42.271635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:42.273026 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.273459 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.273460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.273461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.273525 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:42.273791 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:42.273792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:42.273793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:42.273920 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:42.274137 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.274215 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.274309 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260929 18:55:42.275295 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:55:42.278101 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.275692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.29.71.190:33365)
0929 18:55:42.275740 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.275742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.275848 (+ 106us) server_negotiation.cc:407] Connection header received
0929 18:55:42.275920 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.275920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.275947 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.276013 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:55:42.276111 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.276628 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.277357 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.277545 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.277594 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.277826 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:55:42.277910 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:55:42.277967 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.278032 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.278057 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:55:42.278393 11851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:55:42.280129 11851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:55:42.282367 11965 tablet_service.cc:1467] Processing CreateTablet for tablet cfb2eab27e974628bd33d0e945780ec3 (DEFAULT_TABLE table=test-table [id=08e60167747b4e2baff97aa9189e5cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:42.282562 11965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb2eab27e974628bd33d0e945780ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:42.283880 11931 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Bootstrap starting.
I20260929 18:55:42.284286 11931 tablet_bootstrap.cc:654] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:42.284914 11931 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: No bootstrap required, opened a new log
I20260929 18:55:42.284997 11931 ts_tablet_manager.cc:1397] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 18:55:42.285120 11931 raft_consensus.cc:348] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.285199 11931 raft_consensus.cc:374] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:42.285248 11931 raft_consensus.cc:729] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26291254cd184d37860e029c77ba0e68, State: Initialized, Role: FOLLOWER
I20260929 18:55:42.285310 11931 consensus_queue.cc:260] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.285420 11931 ts_tablet_manager.cc:1428] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.286696 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.282146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:44241 (local address 127.0.0.1:52362)
0929 18:55:42.282309 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.282321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.282375 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.282749 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:42.282750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:42.282753 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:42.282801 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.282803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.284084 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.284085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.285345 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.285348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.286487 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.286488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.286559 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.286564 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.286609 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.286642 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260929 18:55:42.287139 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.282377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.29.71.130:44241)
0929 18:55:42.282559 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.282565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.282577 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:55:42.282624 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.282627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.282647 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.282744 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:42.282886 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.283932 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.285475 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.286371 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.286913 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.287019 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.287047 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.287080 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260929 18:55:42.287377 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.282033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:37295 (local address 127.0.0.1:59818)
0929 18:55:42.282081 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.282092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.282131 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.284131 (+ 2000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:42.284132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:42.284134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:42.284170 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.284171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.285191 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.285192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.286290 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.286293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.287155 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.287156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.287217 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.287220 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.287271 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.287313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:55:42.287846 12111 tablet_service.cc:1467] Processing CreateTablet for tablet cfb2eab27e974628bd33d0e945780ec3 (DEFAULT_TABLE table=test-table [id=08e60167747b4e2baff97aa9189e5cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:42.288162 12111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb2eab27e974628bd33d0e945780ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:42.290400 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.282139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.29.71.131:37295)
0929 18:55:42.283960 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.283965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.283985 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:55:42.284019 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.284021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.284034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.284134 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:42.284225 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.285073 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.286412 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.287052 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.287151 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.287205 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.290250 (+ 3045us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.290301 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1785,"thread_start_us":147,"threads_started":1}
I20260929 18:55:42.291138 12319 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Bootstrap starting.
I20260929 18:55:42.291078 12243 tablet_service.cc:1467] Processing CreateTablet for tablet cfb2eab27e974628bd33d0e945780ec3 (DEFAULT_TABLE table=test-table [id=08e60167747b4e2baff97aa9189e5cfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:42.291399 12243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb2eab27e974628bd33d0e945780ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:42.291900 12319 tablet_bootstrap.cc:654] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:42.292500 12319 log.cc:824] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:42.293308 12319 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: No bootstrap required, opened a new log
I20260929 18:55:42.293429 12319 ts_tablet_manager.cc:1397] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:55:42.294234 12321 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Bootstrap starting.
I20260929 18:55:42.294976 12321 tablet_bootstrap.cc:654] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:42.295539 12321 log.cc:824] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:42.295626 12319 raft_consensus.cc:348] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.295805 12319 raft_consensus.cc:374] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:42.295882 12319 raft_consensus.cc:729] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeaa393ff7c34a7fafcded1577613cad, State: Initialized, Role: FOLLOWER
I20260929 18:55:42.296022 12319 consensus_queue.cc:260] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.296346 12319 ts_tablet_manager.cc:1428] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:55:42.296360 12177 heartbeater.cc:503] Master 127.29.71.190:33365 was elected leader, sending a full tablet report...
I20260929 18:55:42.296646 12321 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: No bootstrap required, opened a new log
I20260929 18:55:42.296733 12321 ts_tablet_manager.cc:1397] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:55:42.298513 12321 raft_consensus.cc:348] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.298666 12321 raft_consensus.cc:374] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:42.298713 12321 raft_consensus.cc:729] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761eefa69d9340b8ad2ccfb7a61b6000, State: Initialized, Role: FOLLOWER
I20260929 18:55:42.298846 12321 consensus_queue.cc:260] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.299129 12321 ts_tablet_manager.cc:1428] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:55:42.299165 12309 heartbeater.cc:503] Master 127.29.71.190:33365 was elected leader, sending a full tablet report...
I20260929 18:55:42.364367 12324 raft_consensus.cc:482] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:42.364491 12324 raft_consensus.cc:504] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.364840 12324 leader_election.cc:288] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241), 26291254cd184d37860e029c77ba0e68 (127.29.71.129:33045)
W20260929 18:55:42.366190 12178 tablet.cc:2369] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:55:42.368311 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.365009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:44241 (local address 127.29.71.131:51203)
0929 18:55:42.365065 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.365077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.365134 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.365295 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:42.365296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:42.365299 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:42.365383 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.365393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.366106 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.366107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.367455 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.367458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.368032 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.368032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.368121 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.368124 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.368193 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.368237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":67}
I20260929 18:55:42.368311 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.365134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:42127 (local address 127.29.71.129:33045)
0929 18:55:42.365180 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.365182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.365294 (+ 112us) server_negotiation.cc:407] Connection header received
0929 18:55:42.365342 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.365343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.365359 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.365411 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:42.365594 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.366130 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.367578 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.367941 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.368029 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.368038 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.368208 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.368251 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:55:42.368332 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.365086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:51203 (local address 127.29.71.130:44241)
0929 18:55:42.365135 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.365137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.365149 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:55:42.365184 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.365185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.365195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.365266 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:42.365466 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.366010 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.367543 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.367938 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.368020 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.368027 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.368207 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.368250 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:55:42.368641 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.365017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33045 (local address 127.29.71.131:42127)
0929 18:55:42.365223 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.365233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.365286 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.365450 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:42.365450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:42.365452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:42.365522 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.365523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.366256 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.366257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.367456 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:42.367458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.368068 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.368069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:42.368123 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.368126 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:55:42.368201 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:55:42.368345 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":30,"thread_start_us":102,"threads_started":1}
I20260929 18:55:42.368780 12131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2eab27e974628bd33d0e945780ec3" candidate_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeaa393ff7c34a7fafcded1577613cad" is_pre_election: true
I20260929 18:55:42.368942 12131 raft_consensus.cc:2454] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761eefa69d9340b8ad2ccfb7a61b6000 in term 0.
I20260929 18:55:42.368966 11986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2eab27e974628bd33d0e945780ec3" candidate_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26291254cd184d37860e029c77ba0e68" is_pre_election: true
I20260929 18:55:42.369107 11986 raft_consensus.cc:2454] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761eefa69d9340b8ad2ccfb7a61b6000 in term 0.
I20260929 18:55:42.369166 12196 leader_election.cc:302] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761eefa69d9340b8ad2ccfb7a61b6000, eeaa393ff7c34a7fafcded1577613cad; no voters:
I20260929 18:55:42.369282 12324 raft_consensus.cc:2781] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:55:42.369335 12324 raft_consensus.cc:482] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:55:42.369390 12324 raft_consensus.cc:3037] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:42.370085 12324 raft_consensus.cc:504] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.370252 12324 leader_election.cc:288] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [CANDIDATE]: Term 1 election: Requested vote from peers eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241), 26291254cd184d37860e029c77ba0e68 (127.29.71.129:33045)
I20260929 18:55:42.370440 11986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2eab27e974628bd33d0e945780ec3" candidate_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26291254cd184d37860e029c77ba0e68"
I20260929 18:55:42.370463 12131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2eab27e974628bd33d0e945780ec3" candidate_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeaa393ff7c34a7fafcded1577613cad"
I20260929 18:55:42.370523 12131 raft_consensus.cc:3037] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:42.370543 11986 raft_consensus.cc:3037] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:42.371258 12131 raft_consensus.cc:2454] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761eefa69d9340b8ad2ccfb7a61b6000 in term 1.
I20260929 18:55:42.371423 12196 leader_election.cc:302] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761eefa69d9340b8ad2ccfb7a61b6000, eeaa393ff7c34a7fafcded1577613cad; no voters:
I20260929 18:55:42.371538 12324 raft_consensus.cc:2781] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:42.371695 11986 raft_consensus.cc:2454] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761eefa69d9340b8ad2ccfb7a61b6000 in term 1.
I20260929 18:55:42.371722 12324 raft_consensus.cc:686] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 LEADER]: Becoming Leader. State: Replica: 761eefa69d9340b8ad2ccfb7a61b6000, State: Running, Role: LEADER
I20260929 18:55:42.371984 12324 consensus_queue.cc:237] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.372880 11850 catalog_manager.cc:5495] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 reported cstate change: term changed from 0 to 1, leader changed from <none> to 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131). New cstate: current_term: 1 leader_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } health_report { overall_health: UNKNOWN } } }
I20260929 18:55:42.378444 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.376210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.29.71.131:37295)
0929 18:55:42.376270 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:42.376272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:42.376308 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:55:42.376333 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:42.376333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:42.376347 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:42.376420 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:55:42.376522 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.377055 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.377735 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:42.377911 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:42.377959 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:42.378080 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:55:42.378208 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:55:42.378275 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:42.378308 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 18:55:42.378325 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:55:42.380053 12131 raft_consensus.cc:1261] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Refusing update from remote peer 761eefa69d9340b8ad2ccfb7a61b6000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:42.380127 11986 raft_consensus.cc:1261] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Refusing update from remote peer 761eefa69d9340b8ad2ccfb7a61b6000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:55:42.380405 12324 consensus_queue.cc:1035] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:42.380546 12326 consensus_queue.cc:1035] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:55:42.382838 12330 mvcc.cc:205] Tried to move back new op lower bound from 7334740551186415616 to 7334740551156449280. Current Snapshot: MvccSnapshot[applied={T|T < 7334740551186415616}]
I20260929 18:55:42.383440 12331 mvcc.cc:205] Tried to move back new op lower bound from 7334740551186415616 to 7334740551156449280. Current Snapshot: MvccSnapshot[applied={T|T < 7334740551186415616 or (T in {7334740551186415616})}]
I20260929 18:55:42.385246 11850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:55:42.385329 11850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:55:42.386071 11850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Sending DeleteTablet for 3 replicas of tablet cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:42.386435 11965 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb2eab27e974628bd33d0e945780ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:42 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:46992
I20260929 18:55:42.386467 12111 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb2eab27e974628bd33d0e945780ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:42 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:52362
I20260929 18:55:42.386536 12048 tablet_replica.cc:330] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: stopping tablet replica
I20260929 18:55:42.386590 12048 raft_consensus.cc:2229] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:55:42.386636 12048 raft_consensus.cc:2258] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:42.386686 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11889
I20260929 18:55:42.387272 12243 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb2eab27e974628bd33d0e945780ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:55:42 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59818
I20260929 18:55:42.387462 12339 tablet_replica.cc:330] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: stopping tablet replica
I20260929 18:55:42.387553 12339 raft_consensus.cc:2229] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:55:42.387676 12339 raft_consensus.cc:2258] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:42.387821 12338 tablet_replica.cc:330] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: stopping tablet replica
I20260929 18:55:42.387925 12338 raft_consensus.cc:2229] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:55:42.388026 12338 raft_consensus.cc:2258] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:55:42.388520 12339 ts_tablet_manager.cc:1905] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:55:42.388553 12338 ts_tablet_manager.cc:1905] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:55:42.390348 12339 ts_tablet_manager.cc:1918] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:55:42.390511 12339 log.cc:1196] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:42.390570 12338 ts_tablet_manager.cc:1918] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:55:42.390645 12338 log.cc:1196] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:42.390942 12338 ts_tablet_manager.cc:1939] T cfb2eab27e974628bd33d0e945780ec3 P eeaa393ff7c34a7fafcded1577613cad: Deleting consensus metadata
I20260929 18:55:42.391026 12339 ts_tablet_manager.cc:1939] T cfb2eab27e974628bd33d0e945780ec3 P 761eefa69d9340b8ad2ccfb7a61b6000: Deleting consensus metadata
I20260929 18:55:42.391397 11835 catalog_manager.cc:4841] TS eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241): tablet cfb2eab27e974628bd33d0e945780ec3 (table test-table [id=08e60167747b4e2baff97aa9189e5cfe]) successfully deleted
I20260929 18:55:42.391564 11835 catalog_manager.cc:4841] TS 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295): tablet cfb2eab27e974628bd33d0e945780ec3 (table test-table [id=08e60167747b4e2baff97aa9189e5cfe]) successfully deleted
W20260929 18:55:42.393718 11837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:33045 (error 108)
I20260929 18:55:42.394073 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 12042
I20260929 18:55:42.394258 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.394089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:33045 (local address 127.0.0.1:46996)
0929 18:55:42.394151 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.394220 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:33045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:55:42.394362 11837 catalog_manager.cc:4584] TS 26291254cd184d37860e029c77ba0e68 (127.29.71.129:33045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfb2eab27e974628bd33d0e945780ec3: Network error: Client connection negotiation failed: client connection to 127.29.71.129:33045: connect: Connection refused (error 111)
I20260929 18:55:42.399580 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 12180
I20260929 18:55:42.404001 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 11818
I20260929 18:55:42.409118 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33365
--webserver_interface=127.29.71.190
--webserver_port=40613
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:42.489421 12341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:42.489658 12341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:42.489692 12341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:42.490866 12341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:42.490954 12341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:42.490979 12341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:42.491009 12341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:42.491026 12341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:42.492473 12341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:42.492734 12341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:42.492952 12341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:42.497824 12346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.497835 12349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.497855 12347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.497982 12341 server_base.cc:1034] running on GCE node
I20260929 18:55:42.498390 12341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:42.498672 12341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:42.499854 12341 hybrid_clock.cc:648] HybridClock initialized: now 1790708142499862 us; error 51 us; skew 500 ppm
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:42.502426 12341 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:42.502915 12341 webserver.cc:460] Webserver started at http://127.29.71.190:40613/ using document root <none> and password file <none>
I20260929 18:55:42.503140 12341 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:42.503207 12341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:42.504484 12341 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.505213 12356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.505412 12341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.505507 12341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "a47ecb1564b5485cb49b5a91477a3503"
format_stamp: "Formatted at 2026-09-29 18:55:40 on dist-test-slave-q2kh"
server_key: "cc7680f9d134798a6ee73a5cdbf0dd1f"
server_key_iv: "9b9ed31232f550acb5abae2f9a8cfa29"
server_key_version: "encryptionkey@0"
I20260929 18:55:42.505833 12341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:42.549449 12341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:42.551904 12341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:42.552066 12341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:42.555975 12341 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:33365
I20260929 18:55:42.556115 12408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:33365 every 8 connection(s)
I20260929 18:55:42.556432 12341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:42.558073 12409 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:55:42.558774 12409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Bootstrap starting.
I20260929 18:55:42.564442 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 12341
I20260929 18:55:42.564785 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:33045
--local_ip_for_outbound_sockets=127.29.71.129
--tserver_master_addrs=127.29.71.190:33365
--webserver_port=36467
--webserver_interface=127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:42.567801 12409 log.cc:824] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:42.570073 12409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:55:42.570418 12409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503: Bootstrap complete.
I20260929 18:55:42.572691 12409 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:42.572921 12409 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47ecb1564b5485cb49b5a91477a3503, State: Initialized, Role: FOLLOWER
I20260929 18:55:42.573100 12409 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:42.573161 12409 raft_consensus.cc:388] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:55:42.573208 12409 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:55:42.573257 12409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:55:42.574299 12409 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:42.574411 12409 leader_election.cc:302] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47ecb1564b5485cb49b5a91477a3503; no voters:
I20260929 18:55:42.574575 12409 leader_election.cc:288] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:55:42.574615 12413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:55:42.574728 12413 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [term 3 LEADER]: Becoming Leader. State: Replica: a47ecb1564b5485cb49b5a91477a3503, State: Running, Role: LEADER
I20260929 18:55:42.574818 12413 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } }
I20260929 18:55:42.574848 12409 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:42.575109 12414 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a47ecb1564b5485cb49b5a91477a3503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } } }
I20260929 18:55:42.575256 12414 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:42.575620 12418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:42.576483 12418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:55:42.576619 12418 catalog_manager.cc:1259] Loaded cluster ID: 79ebc29266274cc1b540ed2ed30d9b8c
I20260929 18:55:42.576661 12418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:55:42.576712 12415 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47ecb1564b5485cb49b5a91477a3503. Latest consensus state: current_term: 3 leader_uuid: "a47ecb1564b5485cb49b5a91477a3503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47ecb1564b5485cb49b5a91477a3503" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 33365 } } }
I20260929 18:55:42.576781 12415 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ecb1564b5485cb49b5a91477a3503 [sys.catalog]: This master's current role is: LEADER
F20260929 18:55:42.576908 12418 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9beab5c9ad google::LogMessage::Fail() at ??:0
@ 0x7f9beab5f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9beab5c4de google::LogMessage::Flush() at ??:0
@ 0x7f9beab5da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9bf2a7d0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9bf2a868b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9beca2a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9beca1f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9bed8836db start_thread at ??:0
@ 0x7f9be965a61f clone at ??:0
W20260929 18:55:42.655848 12411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:42.656111 12411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:42.656152 12411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:42.657370 12411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:42.657440 12411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:42.657487 12411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:55:42.659137 12411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:33045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=36467
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:42.659451 12411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:42.659698 12411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:42.664683 12445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.664695 12448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.664695 12446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.665046 12411 server_base.cc:1034] running on GCE node
I20260929 18:55:42.665241 12411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:42.665505 12411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:42.666675 12411 hybrid_clock.cc:648] HybridClock initialized: now 1790708142666636 us; error 33 us; skew 500 ppm
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:42.669339 12411 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:55:42.669888 12411 webserver.cc:460] Webserver started at http://127.29.71.129:36467/ using document root <none> and password file <none>
I20260929 18:55:42.670197 12411 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:42.670264 12411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:42.671684 12411 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.672402 12455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.672689 12411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.672811 12411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "26291254cd184d37860e029c77ba0e68"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "cf85e43d2915fdb1a3248c7b79d12ce6"
server_key_iv: "6341a23ae72795aab2604cc74f01f027"
server_key_version: "encryptionkey@0"
I20260929 18:55:42.673307 12411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:42.689637 12411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:42.692344 12411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:42.692484 12411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:42.692739 12411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:42.693205 12462 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:55:42.695178 12411 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:55:42.695243 12411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 18:55:42.695291 12411 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:55:42.695966 12411 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:55:42.696024 12411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 18:55:42.696101 12462 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Bootstrap starting.
I20260929 18:55:42.703276 12411 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:33045
I20260929 18:55:42.703439 12569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:33045 every 8 connection(s)
I20260929 18:55:42.703719 12411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:55:42.708303 12462 log.cc:824] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:42.710335 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 12411
I20260929 18:55:42.710638 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:44241
--local_ip_for_outbound_sockets=127.29.71.130
--tserver_master_addrs=127.29.71.190:33365
--webserver_port=37937
--webserver_interface=127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:42.711102 12462 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:55:42.711402 12462 tablet_bootstrap.cc:492] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Bootstrap complete.
I20260929 18:55:42.712622 12462 ts_tablet_manager.cc:1397] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20260929 18:55:42.714040 12462 raft_consensus.cc:348] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.714403 12462 raft_consensus.cc:729] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26291254cd184d37860e029c77ba0e68, State: Initialized, Role: FOLLOWER
I20260929 18:55:42.714589 12462 consensus_queue.cc:260] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:42.714891 12462 ts_tablet_manager.cc:1428] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
W20260929 18:55:42.780532 12572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:55:42.704551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:43037)
0929 18:55:42.704731 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.704782 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:42.705422 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:42.780307 (+ 74885us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
W20260929 18:55:42.780774 12570 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:33365 (0 consecutive failures): Network error: Failed to ping master at 127.29.71.190:33365: Client connection negotiation failed: client connection to 127.29.71.190:33365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:55:42.781150 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.781009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:59521)
0929 18:55:42.781070 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.781120 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:55:42.781617 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.781509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:50315)
0929 18:55:42.781561 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.781590 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:55:42.781739 12570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:55:42.800580 12574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:42.800770 12574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:42.800792 12574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:42.801951 12574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:42.801992 12574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:42.802019 12574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:55:42.803503 12574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:44241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=37937
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:42.803743 12574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:42.803990 12574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:42.809382 12580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.809377 12581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.809456 12574 server_base.cc:1034] running on GCE node
W20260929 18:55:42.809377 12583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.809880 12574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:42.810102 12574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:42.811249 12574 hybrid_clock.cc:648] HybridClock initialized: now 1790708142811234 us; error 27 us; skew 500 ppm
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:42.813704 12574 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:55:42.814203 12574 webserver.cc:460] Webserver started at http://127.29.71.130:37937/ using document root <none> and password file <none>
I20260929 18:55:42.814412 12574 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:42.814455 12574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:42.815687 12574 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.816257 12590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.816422 12574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:55:42.816505 12574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "eeaa393ff7c34a7fafcded1577613cad"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "3275586fedce058469f0f2a4d1d23500"
server_key_iv: "b5375e5a43750a5edce3e3f62ec09d67"
server_key_version: "encryptionkey@0"
I20260929 18:55:42.816811 12574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:42.827467 12574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:42.829555 12574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:42.829656 12574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:42.829813 12574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:42.830045 12574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:42.830075 12574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.830093 12574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:42.830106 12574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.835908 12574 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:44241
I20260929 18:55:42.835981 12703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:44241 every 8 connection(s)
I20260929 18:55:42.836294 12574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:55:42.837293 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.836947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:42959)
0929 18:55:42.837083 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.837155 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260929 18:55:42.837668 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 12574
W20260929 18:55:42.837821 12704 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:33365 (0 consecutive failures): Network error: Failed to ping master at 127.29.71.190:33365: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
I20260929 18:55:42.838006 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:37295
--local_ip_for_outbound_sockets=127.29.71.131
--tserver_master_addrs=127.29.71.190:33365
--webserver_port=38959
--webserver_interface=127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:55:42.838124 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.838009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:58025)
0929 18:55:42.838065 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.838098 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:55:42.838527 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.838365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:43477)
0929 18:55:42.838471 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.838494 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
W20260929 18:55:42.838692 12704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:55:42.920257 12708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:42.920492 12708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:42.920516 12708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:42.921634 12708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:42.921670 12708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:42.921696 12708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:55:42.923113 12708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:37295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=38959
--tserver_master_addrs=127.29.71.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:42.923305 12708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:42.923475 12708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:42.928530 12714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:42.928606 12713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.928656 12708 server_base.cc:1034] running on GCE node
W20260929 18:55:42.928534 12716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:42.929008 12708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:42.929229 12708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:42.930370 12708 hybrid_clock.cc:648] HybridClock initialized: now 1790708142930353 us; error 41 us; skew 500 ppm
Sep 29 18:55:42 dist-test-slave-q2kh krb5kdc[11271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:42.933046 12708 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:55:42.933527 12708 webserver.cc:460] Webserver started at http://127.29.71.131:38959/ using document root <none> and password file <none>
I20260929 18:55:42.933748 12708 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:42.933792 12708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:42.935216 12708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:42.935817 12723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.936020 12708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:55:42.936120 12708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "761eefa69d9340b8ad2ccfb7a61b6000"
format_stamp: "Formatted at 2026-09-29 18:55:41 on dist-test-slave-q2kh"
server_key: "337b9bb5b33d6628f526b5bcf976d4f7"
server_key_iv: "888ba4b9612a9a2ff4a1764323fbb1e8"
server_key_version: "encryptionkey@0"
I20260929 18:55:42.936462 12708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:42.955688 12708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:42.957626 12708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:42.957790 12708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:42.958033 12708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:42.958395 12708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:42.958449 12708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.958490 12708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:42.958519 12708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:42.964426 12708 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:37295
I20260929 18:55:42.964476 12836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:37295 every 8 connection(s)
I20260929 18:55:42.964789 12708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:55:42.965786 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.965440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:45227)
0929 18:55:42.965576 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.965652 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
W20260929 18:55:42.966279 12837 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:33365 (0 consecutive failures): Network error: Failed to ping master at 127.29.71.190:33365: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
I20260929 18:55:42.966560 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.966457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:33717)
0929 18:55:42.966505 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.966541 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:55:42.966804 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:42.966748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:53507)
0929 18:55:42.966776 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:42.966793 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 18:55:42.966931 12837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:55:42.973013 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 12708
I20260929 18:55:43.782460 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:43.782208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:50037)
0929 18:55:43.782358 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:43.782401 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260929 18:55:43.839238 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:43.838986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:38037)
0929 18:55:43.839151 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:43.839187 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260929 18:55:43.967458 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:43.967267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:44375)
0929 18:55:43.967375 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:43.967410 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260929 18:55:43.991547 12844 raft_consensus.cc:482] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:43.991665 12844 raft_consensus.cc:504] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:43.992002 12844 leader_election.cc:288] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295), eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
Sep 29 18:55:43 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.130@KRBTEST.COM
Sep 29 18:55:43 dist-test-slave-q2kh krb5kdc[11271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.131@KRBTEST.COM
I20260929 18:55:44.002731 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:43.992387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:40355 (local address 127.29.71.130:44241)
0929 18:55:43.992516 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:43.992524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:43.992545 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:55:43.993062 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:43.993065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:43.993124 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:43.993274 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:43.993939 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.994808 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.995957 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.996522 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.996633 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:43.998571 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:43.998575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:43.998575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:43.998586 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:44.000515 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:44.001171 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:44.001174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:44.001175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:44.001285 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:44.001636 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:44.001637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:44.001637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:44.001964 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:44.002055 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:44.002494 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 18:55:44.002618 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260929 18:55:44.002731 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:43.992293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:42863 (local address 127.29.71.131:37295)
0929 18:55:43.992454 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:43.992467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:43.992500 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:55:43.993066 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:43.993069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:43.993124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:43.993274 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:43.993937 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.994837 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.995803 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.996416 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.996497 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:43.998993 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:43.998997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:43.998997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:43.999009 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:44.000741 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:44.001171 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:44.001174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:44.001175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:44.001285 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:44.001636 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:44.001637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:44.001637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:44.001964 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:44.002065 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:44.002475 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 18:55:44.002618 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":51,"threads_started":1}
I20260929 18:55:44.002739 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:43.992178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:37295 (local address 127.29.71.129:42863)
0929 18:55:43.992271 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:43.992284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:43.992371 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:43.993310 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:43.993313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:43.993642 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:43.993821 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:43.993826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.994945 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.994945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:43.995715 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:43.995717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.996505 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.996506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:43.996639 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:43.997264 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:43.997269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:43.998899 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:44.000819 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:44.000820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:44.000821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:44.001086 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:44.001366 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:44.001367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:44.001367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:44.001554 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:44.002071 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:44.002080 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:44.002257 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 18:55:44.002405 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 18:55:44.002613 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":46}
I20260929 18:55:44.003341 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:43.992308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:44241 (local address 127.29.71.129:40355)
0929 18:55:43.992424 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:43.992434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:43.992484 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:43.993397 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:43.993399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:43.993630 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:43.993821 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:43.993826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.994899 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.994900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:43.995715 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:43.995717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:43.996611 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:43.996612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:43.996652 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:43.997091 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:43.997098 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:43.998467 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:44.000670 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:44.000676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:44.000678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:44.001059 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:44.001366 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:44.001367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:44.001367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:44.001554 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:44.002071 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:44.002080 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:44.002257 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 18:55:44.002405 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 18:55:44.002670 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":175,"thread_start_us":68,"threads_started":1}
W20260929 18:55:44.003824 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
W20260929 18:55:44.004037 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:44.004096 12456 leader_election.cc:302] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26291254cd184d37860e029c77ba0e68; no voters: 761eefa69d9340b8ad2ccfb7a61b6000, eeaa393ff7c34a7fafcded1577613cad
I20260929 18:55:44.004246 12844 raft_consensus.cc:2726] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:55:44.783236 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:44.782985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:51857)
0929 18:55:44.783138 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:44.783186 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":58,"threads_started":1}
I20260929 18:55:44.839905 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:44.839674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:43203)
0929 18:55:44.839822 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:44.839863 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":54,"threads_started":1}
I20260929 18:55:44.968168 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:44.967958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:53951)
0929 18:55:44.968091 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:44.968129 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260929 18:55:45.783913 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:45.783700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:46863)
0929 18:55:45.783831 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:45.783867 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260929 18:55:45.840584 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:45.840385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:52665)
0929 18:55:45.840503 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:45.840542 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260929 18:55:45.968848 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:45.968615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:53801)
0929 18:55:45.968763 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:45.968803 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260929 18:55:46.118896 12854 raft_consensus.cc:482] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:46.119051 12854 raft_consensus.cc:504] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:46.119258 12854 leader_election.cc:288] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295), eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
W20260929 18:55:46.119719 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
W20260929 18:55:46.119813 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:46.119849 12456 leader_election.cc:302] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26291254cd184d37860e029c77ba0e68; no voters: 761eefa69d9340b8ad2ccfb7a61b6000, eeaa393ff7c34a7fafcded1577613cad
I20260929 18:55:46.119952 12854 raft_consensus.cc:2726] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:55:46.784528 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:46.784311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:34455)
0929 18:55:46.784443 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:46.784481 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":55,"threads_started":1}
I20260929 18:55:46.841207 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:46.841027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:59199)
0929 18:55:46.841138 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:46.841169 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260929 18:55:46.969520 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:46.969325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:55027)
0929 18:55:46.969448 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:46.969480 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260929 18:55:47.785297 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:47.785053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:55821)
0929 18:55:47.785215 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:47.785251 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260929 18:55:47.841892 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:47.841685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:36675)
0929 18:55:47.841807 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:47.841848 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260929 18:55:47.970352 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:47.970102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:41725)
0929 18:55:47.970273 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:47.970313 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260929 18:55:48.498601 12861 raft_consensus.cc:482] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:48.498715 12861 raft_consensus.cc:504] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:48.498906 12861 leader_election.cc:288] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295), eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
W20260929 18:55:48.499293 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
W20260929 18:55:48.499364 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:48.499408 12456 leader_election.cc:302] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26291254cd184d37860e029c77ba0e68; no voters: 761eefa69d9340b8ad2ccfb7a61b6000, eeaa393ff7c34a7fafcded1577613cad
I20260929 18:55:48.499481 12861 raft_consensus.cc:2726] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:55:48.786067 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:48.785748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:44115)
0929 18:55:48.785980 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:48.786021 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":59,"threads_started":1}
I20260929 18:55:48.842617 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:48.842394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:33897)
0929 18:55:48.842532 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:48.842573 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260929 18:55:48.971043 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:48.970829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:45879)
0929 18:55:48.970962 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:48.971000 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260929 18:55:49.786777 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:49.786541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:41069)
0929 18:55:49.786693 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:49.786732 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260929 18:55:49.843348 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:49.843093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:39647)
0929 18:55:49.843241 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:49.843295 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":57,"threads_started":1}
I20260929 18:55:49.971678 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:49.971483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:34773)
0929 18:55:49.971602 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:49.971636 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260929 18:55:50.787480 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:50.787264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:43341)
0929 18:55:50.787389 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:50.787428 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260929 18:55:50.844012 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:50.843800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:47989)
0929 18:55:50.843927 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:50.843960 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260929 18:55:50.972378 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:50.972162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:56665)
0929 18:55:50.972298 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:50.972335 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260929 18:55:51.788300 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:51.788084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:57069)
0929 18:55:51.788217 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:51.788255 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260929 18:55:51.844695 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:51.844498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:33927)
0929 18:55:51.844613 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:51.844650 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260929 18:55:51.973119 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:51.972888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:49339)
0929 18:55:51.973037 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:51.973075 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260929 18:55:52.315480 12874 raft_consensus.cc:482] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:52.315580 12874 raft_consensus.cc:504] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:52.315730 12874 leader_election.cc:288] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295), eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
W20260929 18:55:52.316198 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
W20260929 18:55:52.316260 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:52.316316 12456 leader_election.cc:302] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26291254cd184d37860e029c77ba0e68; no voters: 761eefa69d9340b8ad2ccfb7a61b6000, eeaa393ff7c34a7fafcded1577613cad
I20260929 18:55:52.316393 12874 raft_consensus.cc:2726] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:55:52.789067 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:52.788855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:36397)
0929 18:55:52.788981 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:52.789020 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260929 18:55:52.845443 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:52.845248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:35985)
0929 18:55:52.845365 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:52.845400 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260929 18:55:52.973824 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:52.973609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:56445)
0929 18:55:52.973739 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:52.973779 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260929 18:55:53.789808 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:53.789597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:50965)
0929 18:55:53.789723 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:53.789762 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260929 18:55:53.846333 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:53.846073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:32791)
0929 18:55:53.846219 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:53.846269 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260929 18:55:53.974551 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:53.974348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:33571)
0929 18:55:53.974469 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:53.974505 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260929 18:55:54.790629 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:54.790385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:41409)
0929 18:55:54.790536 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:54.790580 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260929 18:55:54.846972 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:54.846776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:38989)
0929 18:55:54.846886 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:54.846922 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260929 18:55:54.975255 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:54.975020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:53171)
0929 18:55:54.975165 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:54.975208 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260929 18:55:55.791405 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:55.791165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:50669)
0929 18:55:55.791320 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:55.791359 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260929 18:55:55.847667 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:55.847430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:40993)
0929 18:55:55.847571 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:55.847612 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260929 18:55:55.975980 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:55.975766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:53607)
0929 18:55:55.975900 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:55.975936 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260929 18:55:56.792091 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:56.791820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:54601)
0929 18:55:56.791943 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:56.792033 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260929 18:55:56.848412 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:56.848178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:42719)
0929 18:55:56.848324 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:56.848365 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260929 18:55:56.976783 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:56.976537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:48553)
0929 18:55:56.976696 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:56.976738 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260929 18:55:57.120406 12890 raft_consensus.cc:482] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:55:57.120501 12890 raft_consensus.cc:504] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "761eefa69d9340b8ad2ccfb7a61b6000" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 37295 } } peers { permanent_uuid: "eeaa393ff7c34a7fafcded1577613cad" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 44241 } } peers { permanent_uuid: "26291254cd184d37860e029c77ba0e68" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 33045 } }
I20260929 18:55:57.120694 12890 leader_election.cc:288] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295), eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241)
W20260929 18:55:57.121147 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eeaa393ff7c34a7fafcded1577613cad (127.29.71.130:44241): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
W20260929 18:55:57.121236 12456 leader_election.cc:341] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 761eefa69d9340b8ad2ccfb7a61b6000 (127.29.71.131:37295): Not found: Tablet not found: cfb2eab27e974628bd33d0e945780ec3
I20260929 18:55:57.121272 12456 leader_election.cc:302] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26291254cd184d37860e029c77ba0e68; no voters: 761eefa69d9340b8ad2ccfb7a61b6000, eeaa393ff7c34a7fafcded1577613cad
I20260929 18:55:57.121366 12890 raft_consensus.cc:2726] T cfb2eab27e974628bd33d0e945780ec3 P 26291254cd184d37860e029c77ba0e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:55:57.793000 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:57.792736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.129:56617)
0929 18:55:57.792863 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:57.792909 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260929 18:55:57.849153 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:57.848947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.130:54667)
0929 18:55:57.849076 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:57.849109 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260929 18:55:57.977666 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:57.977463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:33365 (local address 127.29.71.131:50903)
0929 18:55:57.977582 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:57.977622 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260929 18:55:57.982637 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 12411
I20260929 18:55:57.987865 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 12574
I20260929 18:55:57.992545 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 12708
2026-09-29T18:55:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18306 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:55:58 dist-test-slave-q2kh krb5kdc[12896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:55:58 dist-test-slave-q2kh krb5kdc[12896](info): set up 2 sockets
Sep 29 18:55:58 dist-test-slave-q2kh krb5kdc[12896](info): commencing operation
krb5kdc: starting...
W20260929 18:55:59.034685 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:55:59.157907 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45471
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:59.237459 12909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:59.237708 12909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:59.237744 12909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:59.238983 12909 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:55:59.239050 12909 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:55:59.239070 12909 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:55:59.239089 12909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:55:59.239122 12909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:59.239153 12909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:59.239176 12909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:55:59.239202 12909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:55:59.240651 12909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:45471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:59.240928 12909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:59.241153 12909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:55:59.246483 12909 server_base.cc:1034] running on GCE node
W20260929 18:55:59.246506 12915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:59.246517 12914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:59.246666 12917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:59.246893 12909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:59.247160 12909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:59.248312 12909 hybrid_clock.cc:648] HybridClock initialized: now 1790708159248275 us; error 55 us; skew 500 ppm
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:59.250839 12909 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:55:59.251300 12909 webserver.cc:460] Webserver started at http://127.29.71.190:45899/ using document root <none> and password file <none>
I20260929 18:55:59.251490 12909 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:59.251530 12909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:59.251618 12909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:59.252641 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "9a253abdb3394bb8a536ea903eee3690"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "3d97d615713ebf66295f6a2b27cc80eb"
server_key_iv: "d5de643d61ab2f8fb8c59c1c30850503"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.252959 12909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "9a253abdb3394bb8a536ea903eee3690"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "3d97d615713ebf66295f6a2b27cc80eb"
server_key_iv: "d5de643d61ab2f8fb8c59c1c30850503"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.254320 12909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:55:59.255036 12924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.255234 12909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:59.255335 12909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "9a253abdb3394bb8a536ea903eee3690"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "3d97d615713ebf66295f6a2b27cc80eb"
server_key_iv: "d5de643d61ab2f8fb8c59c1c30850503"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.255422 12909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:59.262324 12909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:59.264230 12909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:59.264389 12909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:59.268486 12909 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:45471
I20260929 18:55:59.268522 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:45471 every 8 connection(s)
I20260929 18:55:59.268872 12909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:55:59.269838 12977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:59.272178 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Bootstrap starting.
I20260929 18:55:59.272817 12977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:59.272845 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 12909
I20260929 18:55:59.272993 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:55:59.273155 12977 log.cc:824] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:59.273185 29982 external_mini_cluster.cc:1428] Setting key 17bdfc3f5b14954c037540010de6aac1
I20260929 18:55:59.273933 12977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: No bootstrap required, opened a new log
I20260929 18:55:59.276079 12977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:55:59.276240 12977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:59.276289 12977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a253abdb3394bb8a536ea903eee3690, State: Initialized, Role: FOLLOWER
I20260929 18:55:59.276422 12977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:55:59.276513 12977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:55:59.276557 12977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:55:59.276605 12977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:55:59.277158 12977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:55:59.277277 12977 leader_election.cc:302] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a253abdb3394bb8a536ea903eee3690; no voters:
I20260929 18:55:59.277467 12977 leader_election.cc:288] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:55:59.277490 12982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:55:59.277603 12982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 LEADER]: Becoming Leader. State: Replica: 9a253abdb3394bb8a536ea903eee3690, State: Running, Role: LEADER
I20260929 18:55:59.277755 12977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:55:59.277745 12982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:55:59.278344 12984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a253abdb3394bb8a536ea903eee3690. Latest consensus state: current_term: 1 leader_uuid: "9a253abdb3394bb8a536ea903eee3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } } }
I20260929 18:55:59.278440 12984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:59.278873 12983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a253abdb3394bb8a536ea903eee3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } } }
I20260929 18:55:59.278977 12983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: This master's current role is: LEADER
I20260929 18:55:59.279299 12991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:55:59.280014 12991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:59.281634 12991 catalog_manager.cc:1347] Generated new cluster ID: d5e04108f21448f38be0da1bc4c0c091
I20260929 18:55:59.281683 12991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:55:59.284932 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.273711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.29.71.190:45471)
0929 18:55:59.273880 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.273891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.273924 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:55:59.274471 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.274478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.274527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.274672 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:59.275281 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.276161 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.277057 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.277587 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.277693 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.280569 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:59.280572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:59.280573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:59.280584 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:59.283230 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.283651 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.283654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.283654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.283767 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.283971 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.283972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.283972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.284290 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:59.284437 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.284549 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.284704 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260929 18:55:59.302884 12991 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:55:59.303467 12991 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:55:59.305943 12991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Generated new TSK 0
I20260929 18:55:59.306140 12991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:55:59.332796 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:45471
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:59.413403 13003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:59.413599 13003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:59.413620 13003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:59.414743 13003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:59.414783 13003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:59.414808 13003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:55:59.416162 13003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:59.416338 13003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:59.416522 13003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:59.421691 13008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:59.421721 13009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:59.421672 13011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:59.422286 13003 server_base.cc:1034] running on GCE node
I20260929 18:55:59.422540 13003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:59.422794 13003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:59.423964 13003 hybrid_clock.cc:648] HybridClock initialized: now 1790708159423941 us; error 34 us; skew 500 ppm
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:59.426502 13003 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:55:59.426970 13003 webserver.cc:460] Webserver started at http://127.29.71.129:34047/ using document root <none> and password file <none>
I20260929 18:55:59.427217 13003 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:59.427294 13003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:59.427460 13003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:59.428517 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "66f4b26414d74bb09c54b43884d0ec3d"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "0b09637df0738c2e9756355cc75f14c2"
server_key_iv: "a2d5f0e48b835ef27c8b7a826845d0ed"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.428895 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "66f4b26414d74bb09c54b43884d0ec3d"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "0b09637df0738c2e9756355cc75f14c2"
server_key_iv: "a2d5f0e48b835ef27c8b7a826845d0ed"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.430325 13003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:55:59.431092 13018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.431339 13003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:59.431452 13003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "66f4b26414d74bb09c54b43884d0ec3d"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "0b09637df0738c2e9756355cc75f14c2"
server_key_iv: "a2d5f0e48b835ef27c8b7a826845d0ed"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.431540 13003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:59.464468 13003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:59.466671 13003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:59.466854 13003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:59.467103 13003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:59.467424 13003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:59.467482 13003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.467531 13003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:59.467566 13003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.474033 13003 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37413
I20260929 18:55:59.474121 13131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37413 every 8 connection(s)
I20260929 18:55:59.474502 13003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:55:59.477958 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13003
I20260929 18:55:59.478070 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:55:59.478312 29982 external_mini_cluster.cc:1428] Setting key 21234957da59a604bd7c1f76ed753ee8
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:59.484978 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.475332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:54405 (local address 127.29.71.190:45471)
0929 18:55:59.475400 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.475402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.475599 (+ 197us) server_negotiation.cc:407] Connection header received
0929 18:55:59.476106 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.476108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.476118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.476181 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:59.477110 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.477629 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.479187 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.479477 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.479541 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.482029 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:59.482030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:59.482031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:59.482039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:59.483525 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.483998 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.483999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.484000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.484065 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.484326 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.484327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.484327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.484445 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:59.484518 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.484801 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.484888 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:55:59.485173 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.475299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:54405)
0929 18:55:59.475466 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:59.475509 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:59.475999 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:59.476236 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:59.476239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:59.476829 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:59.477018 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.477023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.477788 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.477790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.479104 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.479108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.479573 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.479575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.479731 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.480219 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:59.480229 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:59.481933 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:59.483657 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:59.483664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:59.483666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:59.483910 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:59.484139 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:59.484140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:59.484140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:59.484252 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:59.484519 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:59.484527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:59.484664 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 18:55:59.484774 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:55:59.484930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260929 18:55:59.485622 13132 heartbeater.cc:347] Connected to a master server at 127.29.71.190:45471
I20260929 18:55:59.485702 13132 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:59.485903 13132 heartbeater.cc:511] Master 127.29.71.190:45471 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
I20260929 18:55:59.486541 12941 ts_manager.cc:194] Registered new tserver with Master: 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413)
I20260929 18:55:59.487169 12941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:54405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:55:59.505944 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:45471
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:59.586249 13137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:59.586458 13137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:59.586483 13137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:59.587595 13137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:59.587632 13137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:59.587658 13137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:55:59.589030 13137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:59.589221 13137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:59.589406 13137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:59.594362 13143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:59.594334 13142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:59.594662 13137 server_base.cc:1034] running on GCE node
W20260929 18:55:59.594352 13145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:59.594950 13137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:59.595202 13137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:59.596397 13137 hybrid_clock.cc:648] HybridClock initialized: now 1790708159596379 us; error 27 us; skew 500 ppm
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:59.599272 13137 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:55:59.599692 13137 webserver.cc:460] Webserver started at http://127.29.71.130:46065/ using document root <none> and password file <none>
I20260929 18:55:59.599908 13137 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:59.599969 13137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:59.600117 13137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:59.601249 13137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "a9f117256cfb4aee9f2be9c47f8166f3"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1cc3e846bfbdb5969641cca55bddf1e6"
server_key_iv: "ec58daba3a9001e864b7651a10b27cf0"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.601630 13137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "a9f117256cfb4aee9f2be9c47f8166f3"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1cc3e846bfbdb5969641cca55bddf1e6"
server_key_iv: "ec58daba3a9001e864b7651a10b27cf0"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.603139 13137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:55:59.603992 13152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.604208 13137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:55:59.604296 13137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "a9f117256cfb4aee9f2be9c47f8166f3"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1cc3e846bfbdb5969641cca55bddf1e6"
server_key_iv: "ec58daba3a9001e864b7651a10b27cf0"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.604360 13137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:59.622027 13137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:59.624253 13137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:59.624398 13137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:59.624591 13137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:59.624938 13137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:59.624974 13137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.625036 13137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:59.625067 13137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.631630 13137 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:42549
I20260929 18:55:59.631714 13265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:42549 every 8 connection(s)
I20260929 18:55:59.632050 13137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:59.640924 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13137
I20260929 18:55:59.641041 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:55:59.641321 29982 external_mini_cluster.cc:1428] Setting key 36e9c26c95979fbcbc6be68f71f7dbcc
I20260929 18:55:59.642066 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.632896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:37875 (local address 127.29.71.190:45471)
0929 18:55:59.632978 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.632981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.633110 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:55:59.633627 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.633628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.633638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.633693 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:59.634795 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.635410 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.636487 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.636682 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.636740 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.639081 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:59.639082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:59.639082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:59.639090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:59.640433 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.640850 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.640851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.640852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.640921 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.641268 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.641269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.641269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.641419 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:59.641499 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.641821 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.641939 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:55:59.642277 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.632786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:37875)
0929 18:55:59.632966 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:59.633018 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:59.633542 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:59.633750 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:59.633754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:59.634491 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:59.634709 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.634714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.635488 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.635489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.636404 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.636407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.636758 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.636759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.636878 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.637468 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:59.637477 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:59.638984 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:59.640510 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:59.640516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:59.640518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:59.640753 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:59.641028 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:59.641029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:59.641029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:59.641199 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:59.641501 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:59.641508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:59.641695 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 18:55:59.641789 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:55:59.641963 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260929 18:55:59.643832 13266 heartbeater.cc:347] Connected to a master server at 127.29.71.190:45471
I20260929 18:55:59.643934 13266 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:59.644148 13266 heartbeater.cc:511] Master 127.29.71.190:45471 requested a full tablet report, sending...
I20260929 18:55:59.644555 12941 ts_manager.cc:194] Registered new tserver with Master: a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549)
I20260929 18:55:59.644970 12941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:37875
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:55:59.669545 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:45471
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:55:59.750514 13271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:55:59.750766 13271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:55:59.750806 13271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:55:59.751962 13271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:55:59.752027 13271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:55:59.752071 13271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:55:59.753513 13271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:55:59.753778 13271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:55:59.754000 13271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:55:59.759241 13277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:55:59.759272 13276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:59.759600 13271 server_base.cc:1034] running on GCE node
W20260929 18:55:59.759265 13279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:55:59.759871 13271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:55:59.760195 13271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:55:59.761368 13271 hybrid_clock.cc:648] HybridClock initialized: now 1790708159761354 us; error 32 us; skew 500 ppm
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:55:59.764040 13271 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:55:59.764537 13271 webserver.cc:460] Webserver started at http://127.29.71.131:36611/ using document root <none> and password file <none>
I20260929 18:55:59.764780 13271 fs_manager.cc:359] Metadata directory not provided
I20260929 18:55:59.764863 13271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:55:59.765019 13271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:55:59.766141 13271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1016334e61d6f29b7ee02590f45bf25c"
server_key_iv: "3b890ca1a2b47ae1670d5620b5108ab6"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.766630 13271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1016334e61d6f29b7ee02590f45bf25c"
server_key_iv: "3b890ca1a2b47ae1670d5620b5108ab6"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.768033 13271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:55:59.768906 13286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.769073 13271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:55:59.769184 13271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1016334e61d6f29b7ee02590f45bf25c"
server_key_iv: "3b890ca1a2b47ae1670d5620b5108ab6"
server_key_version: "encryptionkey@0"
I20260929 18:55:59.769263 13271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:55:59.792694 13271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:55:59.795065 13271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:55:59.795228 13271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:55:59.795506 13271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:55:59.795854 13271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:55:59.795914 13271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.795961 13271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:55:59.795990 13271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:55:59.802086 13271 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:32933
I20260929 18:55:59.802181 13399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:32933 every 8 connection(s)
I20260929 18:55:59.802528 13271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:55:59.804788 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13271
I20260929 18:55:59.804905 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:55:59.805147 29982 external_mini_cluster.cc:1428] Setting key 3a3c19644bfcd8b154ca0fbade71d876
Sep 29 18:55:59 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708159, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:55:59.812189 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.803296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:34767 (local address 127.29.71.190:45471)
0929 18:55:59.803378 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.803381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.803568 (+ 187us) server_negotiation.cc:407] Connection header received
0929 18:55:59.804097 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.804098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.804112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.804175 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:59.805151 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.806065 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.807101 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.807303 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.807362 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.809591 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:59.809592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:59.809592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:59.809599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:59.810959 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.811333 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.811334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.811335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.811388 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.811600 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.811601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.811601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.811702 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:59.811762 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.812051 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.812124 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:55:59.812419 13402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.803260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:34767)
0929 18:55:59.803446 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:59.803480 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:59.803978 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:59.804228 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:59.804231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:59.804787 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:55:59.805050 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.805062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.806191 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.806192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.807028 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.807031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.807378 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.807379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.807474 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.807999 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:55:59.808016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:55:59.809507 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:55:59.811047 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:59.811053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:59.811054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:59.811262 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:59.811451 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:55:59.811451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:55:59.811452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:55:59.811543 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:55:59.811759 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:55:59.811763 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:55:59.811907 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 18:55:59.811991 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:55:59.812105 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260929 18:55:59.813002 13400 heartbeater.cc:347] Connected to a master server at 127.29.71.190:45471
I20260929 18:55:59.813089 13400 heartbeater.cc:464] Registering TS with master...
I20260929 18:55:59.813299 13400 heartbeater.cc:511] Master 127.29.71.190:45471 requested a full tablet report, sending...
I20260929 18:55:59.813694 12941 ts_manager.cc:194] Registered new tserver with Master: a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
I20260929 18:55:59.814114 12941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:34767
I20260929 18:55:59.817251 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:55:59.824728 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.818705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.29.71.190:45471)
0929 18:55:59.818771 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.818773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.818907 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:55:59.818954 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.818955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.818963 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.819010 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:55:59.819568 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.820044 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.820793 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.820990 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.821043 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.822347 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:55:59.822349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:55:59.822349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:55:59.822355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:55:59.823714 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.824029 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.824030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.824030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.824080 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:55:59.824271 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:55:59.824271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:55:59.824272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:55:59.824399 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:55:59.824474 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.824585 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.824643 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 18:55:59.826442 12941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:55:59.828102 12941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:55:59.834987 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.830804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.29.71.131:32933)
0929 18:55:59.831022 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.831028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.831040 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:55:59.831083 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.831086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.831103 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.831485 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:59.831555 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.832761 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.833970 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.834594 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.834670 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.834723 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.834856 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.834896 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260929 18:55:59.835022 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.830575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37413 (local address 127.0.0.1:40724)
0929 18:55:59.830738 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:59.830778 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:59.830844 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:59.831194 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:59.831197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:59.831199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:59.831266 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.831267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.832361 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.832362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.833465 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.833468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.834613 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.834614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.834739 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.834760 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:55:59.834853 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:55:59.834915 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":26,"thread_start_us":42,"threads_started":1}
I20260929 18:55:59.835022 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.830589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:32933 (local address 127.0.0.1:35492)
0929 18:55:59.830738 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:59.830778 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:59.830844 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:59.831299 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:59.831300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:59.831303 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:59.831389 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.831392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.832878 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.832879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.833887 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.833889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.834681 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.834682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.834739 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.834759 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:55:59.834843 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:55:59.834915 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260929 18:55:59.835330 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.830870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.29.71.129:37413)
0929 18:55:59.831047 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.831051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.831059 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:55:59.831090 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.831091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.831103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.831190 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:59.831349 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.832261 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.833541 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.834434 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.834512 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.834577 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.835190 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.835238 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260929 18:55:59.836109 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.830589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42549 (local address 127.0.0.1:42588)
0929 18:55:59.831339 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 18:55:59.831350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:55:59.831434 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:55:59.831664 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:55:59.831665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:55:59.831667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:55:59.831711 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.831713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.833124 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.833125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.834468 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:55:59.834471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.835840 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.835841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:55:59.835908 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.835912 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:55:59.835987 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:55:59.836034 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":104,"threads_started":1}
I20260929 18:55:59.836263 13334 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9d4fd3fae04cae8e1b82eb18cde610 (DEFAULT_TABLE table=test-table [id=57616c213a4c4bdb883f228f680c6046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:59.836283 13066 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9d4fd3fae04cae8e1b82eb18cde610 (DEFAULT_TABLE table=test-table [id=57616c213a4c4bdb883f228f680c6046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:59.836426 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:55:59.831085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.29.71.130:42549)
0929 18:55:59.831292 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 18:55:59.831296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:55:59.831405 (+ 109us) server_negotiation.cc:407] Connection header received
0929 18:55:59.831548 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:55:59.831550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:55:59.831566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:55:59.831655 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:55:59.831888 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.833020 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.834555 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:55:59.835345 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:55:59.835418 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:55:59.835474 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:55:59.836287 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 18:55:59.836347 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260929 18:55:59.836674 13334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9d4fd3fae04cae8e1b82eb18cde610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:59.836673 13066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9d4fd3fae04cae8e1b82eb18cde610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:59.837088 13200 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9d4fd3fae04cae8e1b82eb18cde610 (DEFAULT_TABLE table=test-table [id=57616c213a4c4bdb883f228f680c6046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:55:59.837378 13200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9d4fd3fae04cae8e1b82eb18cde610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:55:59.839331 13417 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Bootstrap starting.
I20260929 18:55:59.839956 13417 tablet_bootstrap.cc:654] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:59.840198 13418 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap starting.
I20260929 18:55:59.840564 13417 log.cc:824] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:59.840739 13419 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Bootstrap starting.
I20260929 18:55:59.840929 13418 tablet_bootstrap.cc:654] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:59.841387 13419 tablet_bootstrap.cc:654] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:55:59.841526 13418 log.cc:824] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:59.841565 13417 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: No bootstrap required, opened a new log
I20260929 18:55:59.841625 13417 ts_tablet_manager.cc:1397] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:55:59.842024 13419 log.cc:824] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:55:59.842412 13418 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: No bootstrap required, opened a new log
I20260929 18:55:59.842499 13418 ts_tablet_manager.cc:1397] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:55:59.842729 13419 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: No bootstrap required, opened a new log
I20260929 18:55:59.842986 13419 ts_tablet_manager.cc:1397] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:55:59.843850 13417 raft_consensus.cc:348] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:55:59.844043 13417 raft_consensus.cc:374] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:59.844081 13417 raft_consensus.cc:729] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a4715b594e4f4f92bf7ef5b1a0c817, State: Initialized, Role: FOLLOWER
I20260929 18:55:59.844209 13417 consensus_queue.cc:260] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:55:59.844511 13417 ts_tablet_manager.cc:1428] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:55:59.844696 13400 heartbeater.cc:503] Master 127.29.71.190:45471 was elected leader, sending a full tablet report...
I20260929 18:55:59.844728 13419 raft_consensus.cc:348] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:55:59.844946 13419 raft_consensus.cc:374] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:59.844874 13418 raft_consensus.cc:348] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:55:59.844998 13418 raft_consensus.cc:374] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:55:59.844988 13419 raft_consensus.cc:729] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f117256cfb4aee9f2be9c47f8166f3, State: Initialized, Role: FOLLOWER
I20260929 18:55:59.845021 13418 raft_consensus.cc:729] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f4b26414d74bb09c54b43884d0ec3d, State: Initialized, Role: FOLLOWER
I20260929 18:55:59.845103 13419 consensus_queue.cc:260] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:55:59.845175 13418 consensus_queue.cc:260] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:55:59.845346 13419 ts_tablet_manager.cc:1428] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:55:59.845469 13266 heartbeater.cc:503] Master 127.29.71.190:45471 was elected leader, sending a full tablet report...
I20260929 18:55:59.845460 13418 ts_tablet_manager.cc:1428] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:55:59.845584 13132 heartbeater.cc:503] Master 127.29.71.190:45471 was elected leader, sending a full tablet report...
W20260929 18:55:59.882623 13267 tablet.cc:2369] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:55:59.975272 13133 tablet.cc:2369] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:56:00.053141 13401 tablet.cc:2369] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:56:00.060871 13424 raft_consensus.cc:482] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:00.060994 13424 raft_consensus.cc:504] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:56:00.061473 13424 leader_election.cc:288] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413), a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
I20260929 18:56:00.064409 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:00.061700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:45787 (local address 127.29.71.129:37413)
0929 18:56:00.061754 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:00.061756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:00.061796 (+ 40us) server_negotiation.cc:407] Connection header received
0929 18:56:00.061834 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:00.061835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:00.061845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:00.061922 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:00.062102 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.062682 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.063643 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.064016 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.064111 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:00.064120 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:00.064296 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 18:56:00.064332 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 18:56:00.064440 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:00.061638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37413 (local address 127.29.71.130:45787)
0929 18:56:00.061692 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:00.061704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:00.061749 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:00.061946 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:00.061947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:00.061950 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:00.062025 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:00.062026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.062798 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.062799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:00.063539 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:00.063542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.064110 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.064111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:00.064197 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:00.064200 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:56:00.064267 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:56:00.064315 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:56:00.064455 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:00.061753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:33677 (local address 127.29.71.131:32933)
0929 18:56:00.061804 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:00.061806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:00.061971 (+ 165us) server_negotiation.cc:407] Connection header received
0929 18:56:00.062043 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:00.062044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:00.062059 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:00.062119 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:00.062325 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.062772 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.063700 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.064080 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.064164 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:00.064172 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:00.064344 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 18:56:00.064383 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:56:00.064556 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:00.061660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:32933 (local address 127.29.71.130:33677)
0929 18:56:00.061899 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:00.061908 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:00.061975 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:00.062122 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:00.062123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:00.062124 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:00.062230 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:00.062232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.062861 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.062862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:00.063606 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:00.063608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.064166 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.064167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:00.064253 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:00.064256 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:56:00.064319 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:56:00.064360 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":130,"threads_started":1}
I20260929 18:56:00.064826 13086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9d4fd3fae04cae8e1b82eb18cde610" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f4b26414d74bb09c54b43884d0ec3d" is_pre_election: true
I20260929 18:56:00.064981 13086 raft_consensus.cc:2454] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 0.
I20260929 18:56:00.065111 13354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9d4fd3fae04cae8e1b82eb18cde610" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" is_pre_election: true
I20260929 18:56:00.065208 13154 leader_election.cc:302] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d, a9f117256cfb4aee9f2be9c47f8166f3; no voters:
I20260929 18:56:00.065261 13354 raft_consensus.cc:2454] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 0.
I20260929 18:56:00.065343 13424 raft_consensus.cc:2781] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:56:00.065445 13424 raft_consensus.cc:482] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:56:00.065477 13424 raft_consensus.cc:3037] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:00.066226 13424 raft_consensus.cc:504] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:56:00.066382 13424 leader_election.cc:288] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 election: Requested vote from peers 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413), a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
I20260929 18:56:00.066521 13086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9d4fd3fae04cae8e1b82eb18cde610" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f4b26414d74bb09c54b43884d0ec3d"
I20260929 18:56:00.066524 13354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9d4fd3fae04cae8e1b82eb18cde610" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817"
I20260929 18:56:00.066602 13086 raft_consensus.cc:3037] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:00.066620 13354 raft_consensus.cc:3037] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:00.067324 13086 raft_consensus.cc:2454] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 1.
I20260929 18:56:00.067330 13354 raft_consensus.cc:2454] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 1.
I20260929 18:56:00.067495 13154 leader_election.cc:302] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d, a9f117256cfb4aee9f2be9c47f8166f3; no voters:
I20260929 18:56:00.067642 13424 raft_consensus.cc:2781] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:56:00.067818 13424 raft_consensus.cc:686] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 LEADER]: Becoming Leader. State: Replica: a9f117256cfb4aee9f2be9c47f8166f3, State: Running, Role: LEADER
I20260929 18:56:00.067920 13424 consensus_queue.cc:237] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:56:00.068747 12941 catalog_manager.cc:5495] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130). New cstate: current_term: 1 leader_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } health_report { overall_health: UNKNOWN } } }
I20260929 18:56:00.119640 13416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:00.116900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.29.71.130:42549)
0929 18:56:00.116967 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:00.116970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:00.116990 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:56:00.117044 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:00.117044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:00.117063 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:00.117147 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:56:00.117331 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.117841 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.118755 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:00.118952 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:00.119041 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:00.119230 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:56:00.119346 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:56:00.119455 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:00.119499 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 18:56:00.119521 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:56:00.121523 13086 raft_consensus.cc:1261] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Refusing update from remote peer a9f117256cfb4aee9f2be9c47f8166f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:00.121544 13354 raft_consensus.cc:1261] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Refusing update from remote peer a9f117256cfb4aee9f2be9c47f8166f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:00.121927 13427 consensus_queue.cc:1035] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:00.122081 13424 consensus_queue.cc:1035] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:00.124686 13430 mvcc.cc:205] Tried to move back new op lower bound from 7334740623854870528 to 7334740623638822912. Current Snapshot: MvccSnapshot[applied={T|T < 7334740623854870528}]
I20260929 18:56:00.125638 13431 mvcc.cc:205] Tried to move back new op lower bound from 7334740623854870528 to 7334740623638822912. Current Snapshot: MvccSnapshot[applied={T|T < 7334740623854870528 or (T in {7334740623854870528})}]
I20260929 18:56:00.126870 12940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:00.126967 12940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:00.127892 12940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Sending DeleteTablet for 3 replicas of tablet 8f9d4fd3fae04cae8e1b82eb18cde610
I20260929 18:56:00.128249 13200 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9d4fd3fae04cae8e1b82eb18cde610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:00 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:42588
I20260929 18:56:00.128330 13334 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9d4fd3fae04cae8e1b82eb18cde610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:00 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:35492
I20260929 18:56:00.128490 13440 tablet_replica.cc:330] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: stopping tablet replica
I20260929 18:56:00.128545 13441 tablet_replica.cc:330] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: stopping tablet replica
I20260929 18:56:00.128551 13066 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9d4fd3fae04cae8e1b82eb18cde610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:00 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:40724
I20260929 18:56:00.128584 13440 raft_consensus.cc:2229] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:56:00.128641 13441 raft_consensus.cc:2229] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:56:00.128669 13440 raft_consensus.cc:2258] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:00.128747 13441 raft_consensus.cc:2258] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:00.128809 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 13003
I20260929 18:56:00.129112 13440 ts_tablet_manager.cc:1905] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:56:00.131181 13440 ts_tablet_manager.cc:1918] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:00.131305 13440 log.cc:1196] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/8f9d4fd3fae04cae8e1b82eb18cde610
I20260929 18:56:00.131338 13441 ts_tablet_manager.cc:1905] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:56:00.131623 13440 ts_tablet_manager.cc:1939] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting consensus metadata
I20260929 18:56:00.132043 12925 catalog_manager.cc:4841] TS a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549): tablet 8f9d4fd3fae04cae8e1b82eb18cde610 (table test-table [id=57616c213a4c4bdb883f228f680c6046]) successfully deleted
I20260929 18:56:00.132977 13441 ts_tablet_manager.cc:1918] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:00.133052 13441 log.cc:1196] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/8f9d4fd3fae04cae8e1b82eb18cde610
I20260929 18:56:00.133416 13441 ts_tablet_manager.cc:1939] T 8f9d4fd3fae04cae8e1b82eb18cde610 P a4a4715b594e4f4f92bf7ef5b1a0c817: Deleting consensus metadata
I20260929 18:56:00.133829 12927 catalog_manager.cc:4841] TS a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933): tablet 8f9d4fd3fae04cae8e1b82eb18cde610 (table test-table [id=57616c213a4c4bdb883f228f680c6046]) successfully deleted
W20260929 18:56:00.136253 12926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:37413 (error 108)
I20260929 18:56:00.136446 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 13137
I20260929 18:56:00.136749 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:00.136640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37413 (local address 127.0.0.1:40740)
0929 18:56:00.136687 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:00.136714 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:56:00.137079 12926 catalog_manager.cc:4584] TS 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f9d4fd3fae04cae8e1b82eb18cde610: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37413: connect: Connection refused (error 111)
I20260929 18:56:00.142627 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 13271
I20260929 18:56:00.147257 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 12909
I20260929 18:56:20.153225 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45471
--webserver_interface=127.29.71.190
--webserver_port=45899
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:56:20.243158 13444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:20.243448 13444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:20.243494 13444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:20.244736 13444 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:56:20.244808 13444 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:56:20.244831 13444 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:56:20.244851 13444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:56:20.244880 13444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:20.244906 13444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:20.244932 13444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:56:20.244961 13444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:56:20.246549 13444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:45471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=45899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:20.246845 13444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:20.247087 13444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:56:20.256381 13444 server_base.cc:1034] running on GCE node
W20260929 18:56:20.256317 13450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.256397 13451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.256364 13453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:20.256832 13444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:20.257169 13444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:20.258329 13444 hybrid_clock.cc:648] HybridClock initialized: now 1790708180258304 us; error 60 us; skew 500 ppm
Sep 29 18:56:20 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:20.266386 13444 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:56:20.266836 13444 webserver.cc:460] Webserver started at http://127.29.71.190:45899/ using document root <none> and password file <none>
I20260929 18:56:20.267103 13444 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:20.267160 13444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:20.268703 13444 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:20.269429 13460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.269688 13444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:20.269796 13444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "9a253abdb3394bb8a536ea903eee3690"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "3d97d615713ebf66295f6a2b27cc80eb"
server_key_iv: "d5de643d61ab2f8fb8c59c1c30850503"
server_key_version: "encryptionkey@0"
I20260929 18:56:20.270265 13444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:20.278818 13444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:20.280862 13444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:20.281005 13444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:20.285187 13444 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:45471
I20260929 18:56:20.285218 13512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:45471 every 8 connection(s)
I20260929 18:56:20.285612 13444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:56:20.287168 13513 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:56:20.287835 13513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Bootstrap starting.
I20260929 18:56:20.288731 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13444
I20260929 18:56:20.289014 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:37413
--local_ip_for_outbound_sockets=127.29.71.129
--tserver_master_addrs=127.29.71.190:45471
--webserver_port=34047
--webserver_interface=127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:56:20.296564 13513 log.cc:824] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:20.298571 13513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:56:20.298866 13513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Bootstrap complete.
I20260929 18:56:20.301084 13513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:20.301322 13513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a253abdb3394bb8a536ea903eee3690, State: Initialized, Role: FOLLOWER
I20260929 18:56:20.301501 13513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:20.301576 13513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:56:20.301630 13513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:56:20.301679 13513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:56:20.302793 13513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:20.302906 13513 leader_election.cc:302] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a253abdb3394bb8a536ea903eee3690; no voters:
I20260929 18:56:20.303087 13513 leader_election.cc:288] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:56:20.303128 13517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:56:20.303242 13517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 LEADER]: Becoming Leader. State: Replica: 9a253abdb3394bb8a536ea903eee3690, State: Running, Role: LEADER
I20260929 18:56:20.303354 13513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:56:20.303392 13517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:20.303694 13517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a253abdb3394bb8a536ea903eee3690. Latest consensus state: current_term: 2 leader_uuid: "9a253abdb3394bb8a536ea903eee3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } } }
I20260929 18:56:20.303699 13519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a253abdb3394bb8a536ea903eee3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } } }
I20260929 18:56:20.303876 13519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:20.303865 13517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:20.304111 13524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:56:20.305008 13524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:56:20.305153 13524 catalog_manager.cc:1259] Loaded cluster ID: d5e04108f21448f38be0da1bc4c0c091
I20260929 18:56:20.305395 13524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:56:20.306589 13524 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:56:20.306763 13524 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Loaded TSK: 0
I20260929 18:56:20.320227 13524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Generated new TSK 1
I20260929 18:56:20.320343 13524 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:56:20.373435 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:20.373769 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:20.373813 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:20.375313 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:20.375378 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:20.375425 13515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:56:20.376873 13515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:37413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=34047
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:20.377111 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:20.377350 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:20.382375 13540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.382359 13541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.382375 13543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:20.382925 13515 server_base.cc:1034] running on GCE node
I20260929 18:56:20.383147 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:20.383386 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:20.384541 13515 hybrid_clock.cc:648] HybridClock initialized: now 1790708180384527 us; error 31 us; skew 500 ppm
Sep 29 18:56:20 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:20.387333 13515 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:56:20.387856 13515 webserver.cc:460] Webserver started at http://127.29.71.129:34047/ using document root <none> and password file <none>
I20260929 18:56:20.388087 13515 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:20.388160 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:20.389567 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:20.390276 13550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.390439 13515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:20.390542 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "66f4b26414d74bb09c54b43884d0ec3d"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "0b09637df0738c2e9756355cc75f14c2"
server_key_iv: "a2d5f0e48b835ef27c8b7a826845d0ed"
server_key_version: "encryptionkey@0"
I20260929 18:56:20.390899 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:20.400964 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:20.403290 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:20.403455 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:20.403723 13515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:20.404258 13557 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:56:20.405803 13515 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:56:20.405864 13515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 18:56:20.405898 13515 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:56:20.406637 13515 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:56:20.406682 13515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 18:56:20.406724 13557 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap starting.
I20260929 18:56:20.414503 13515 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37413
I20260929 18:56:20.415045 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:56:20.415452 13664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37413 every 8 connection(s)
I20260929 18:56:20.418680 13557 log.cc:824] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:20.420912 13557 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:56:20.421224 13557 tablet_bootstrap.cc:492] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap complete.
I20260929 18:56:20.422394 13557 ts_tablet_manager.cc:1397] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260929 18:56:20.424307 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13515
I20260929 18:56:20.424539 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:42549
--local_ip_for_outbound_sockets=127.29.71.130
--tserver_master_addrs=127.29.71.190:45471
--webserver_port=46065
--webserver_interface=127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:56:20 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:56:20.423877 13557 raft_consensus.cc:348] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:56:20.426000 13557 raft_consensus.cc:729] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f4b26414d74bb09c54b43884d0ec3d, State: Initialized, Role: FOLLOWER
I20260929 18:56:20.426199 13557 consensus_queue.cc:260] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } }
I20260929 18:56:20.426513 13557 ts_tablet_manager.cc:1428] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:56:20.429945 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.416177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:49079 (local address 127.29.71.190:45471)
0929 18:56:20.416433 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.416442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.416491 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:56:20.417437 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.417444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.417501 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.417645 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:20.418799 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.420302 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.421905 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.422613 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.422730 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.425503 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:20.425507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:20.425507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:20.425520 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:20.427879 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.428375 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.428378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.428378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.428485 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.428812 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.428813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.428813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.429117 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:20.429210 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.429588 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.429721 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260929 18:56:20.430006 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.415774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:49079)
0929 18:56:20.415994 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.416047 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.417179 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.417719 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.417724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.418281 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:20.418618 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.418626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.420442 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.420444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.421790 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.421793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.422717 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.422719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.422873 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.423517 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:20.423534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:20.425386 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:20.427982 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:20.427989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:20.427992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:20.428284 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:20.428570 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:20.428572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:20.428572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:20.428730 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:20.429198 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:20.429207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:20.429391 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.429525 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.429730 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260929 18:56:20.431267 13665 heartbeater.cc:347] Connected to a master server at 127.29.71.190:45471
I20260929 18:56:20.431383 13665 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:20.431598 13665 heartbeater.cc:511] Master 127.29.71.190:45471 requested a full tablet report, sending...
I20260929 18:56:20.432451 13477 ts_manager.cc:194] Registered new tserver with Master: 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413)
I20260929 18:56:20.433642 13477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:49079
I20260929 18:56:20.434737 13665 heartbeater.cc:503] Master 127.29.71.190:45471 was elected leader, sending a full tablet report...
I20260929 18:56:20.438627 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.433460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.29.71.129:37413)
0929 18:56:20.434497 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.434501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.434509 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:56:20.434540 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.434542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.434556 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.434796 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:20.434842 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.435911 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.437485 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.438221 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.438289 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.438340 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.438515 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.438556 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":73,"threads_started":1}
I20260929 18:56:20.438628 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.433179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37413 (local address 127.0.0.1:33260)
0929 18:56:20.434275 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.434311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.434375 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.434644 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.434647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.434648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:20.434721 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.434724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.436009 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.436010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.437389 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.437392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.438313 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.438315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.438409 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.438424 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.438493 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.438547 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":97,"threads_started":1}
I20260929 18:56:20.439368 13587 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9d4fd3fae04cae8e1b82eb18cde610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:00 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:33260
I20260929 18:56:20.439951 13674 tablet_replica.cc:330] stopping tablet replica
I20260929 18:56:20.440054 13674 raft_consensus.cc:2229] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:56:20.440116 13674 raft_consensus.cc:2258] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:20.440416 13674 ts_tablet_manager.cc:1905] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:56:20.442972 13674 ts_tablet_manager.cc:1918] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:20.443049 13674 log.cc:1196] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal/wals/8f9d4fd3fae04cae8e1b82eb18cde610
I20260929 18:56:20.443279 13674 ts_tablet_manager.cc:1939] T 8f9d4fd3fae04cae8e1b82eb18cde610 P 66f4b26414d74bb09c54b43884d0ec3d: Deleting consensus metadata
I20260929 18:56:20.443800 13462 catalog_manager.cc:4841] TS 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413): tablet 8f9d4fd3fae04cae8e1b82eb18cde610 (table test-table [id=57616c213a4c4bdb883f228f680c6046]) successfully deleted
W20260929 18:56:20.509158 13670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:20.509387 13670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:20.509425 13670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:20.510562 13670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:20.510622 13670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:20.510667 13670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:56:20.512208 13670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=46065
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:20.512475 13670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:20.512729 13670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:20.517669 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.517681 13682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.517737 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:20.517877 13670 server_base.cc:1034] running on GCE node
I20260929 18:56:20.518061 13670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:20.518322 13670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:20.519479 13670 hybrid_clock.cc:648] HybridClock initialized: now 1790708180519472 us; error 35 us; skew 500 ppm
Sep 29 18:56:20 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:20.522012 13670 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:56:20.522485 13670 webserver.cc:460] Webserver started at http://127.29.71.130:46065/ using document root <none> and password file <none>
I20260929 18:56:20.522723 13670 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:20.522800 13670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:20.524117 13670 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:56:20.524683 13689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.524878 13670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:56:20.524972 13670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "a9f117256cfb4aee9f2be9c47f8166f3"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1cc3e846bfbdb5969641cca55bddf1e6"
server_key_iv: "ec58daba3a9001e864b7651a10b27cf0"
server_key_version: "encryptionkey@0"
I20260929 18:56:20.525331 13670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:20.551777 13670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:20.554129 13670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:20.554318 13670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:20.554554 13670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:20.554867 13670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:20.554917 13670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.554962 13670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:20.554991 13670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.560756 13670 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:42549
I20260929 18:56:20.560797 13802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:42549 every 8 connection(s)
I20260929 18:56:20.561141 13670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:56:20 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:56:20.569628 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.561843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:60295 (local address 127.29.71.190:45471)
0929 18:56:20.561911 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.561913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.562187 (+ 274us) server_negotiation.cc:407] Connection header received
0929 18:56:20.562703 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.562704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.562718 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.562772 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:20.563665 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.564159 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.565088 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.565274 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.565325 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.567365 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:20.567366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:20.567372 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:20.567379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:20.568530 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.568895 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.568896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.568896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.568941 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.569140 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.569141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.569141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.569236 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:20.569284 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.569499 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.569563 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:56:20.569783 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.561844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:60295)
0929 18:56:20.562052 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.562089 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.562610 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.562825 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.562829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.563385 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:20.563578 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.563583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.564235 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.564236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.565017 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.565019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.565342 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.565343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.565443 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.565955 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:20.565963 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:20.567287 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:20.568595 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:20.568601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:20.568603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:20.568806 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:20.568989 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:20.568990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:20.568990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:20.569091 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:20.569285 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:20.569288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:20.569399 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.569478 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.569590 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260929 18:56:20.570044 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13670
I20260929 18:56:20.570314 13803 heartbeater.cc:347] Connected to a master server at 127.29.71.190:45471
I20260929 18:56:20.570426 13803 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:20.570415 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:32933
--local_ip_for_outbound_sockets=127.29.71.131
--tserver_master_addrs=127.29.71.190:45471
--webserver_port=36611
--webserver_interface=127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:56:20.570631 13803 heartbeater.cc:511] Master 127.29.71.190:45471 requested a full tablet report, sending...
I20260929 18:56:20.571024 13477 ts_manager.cc:194] Registered new tserver with Master: a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549)
I20260929 18:56:20.571411 13477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:60295
W20260929 18:56:20.651031 13806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:20.651240 13806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:20.651263 13806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:20.652462 13806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:20.652524 13806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:20.652570 13806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:56:20.654052 13806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:32933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=36611
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:20.654340 13806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:20.654568 13806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:20.659514 13811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.659499 13812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:20.659570 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:20.659683 13806 server_base.cc:1034] running on GCE node
I20260929 18:56:20.660004 13806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:20.660255 13806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:20.661420 13806 hybrid_clock.cc:648] HybridClock initialized: now 1790708180661404 us; error 31 us; skew 500 ppm
Sep 29 18:56:20 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:20.663960 13806 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:56:20.664402 13806 webserver.cc:460] Webserver started at http://127.29.71.131:36611/ using document root <none> and password file <none>
I20260929 18:56:20.664593 13806 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:20.664644 13806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:20.665998 13806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:20.666610 13821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.666770 13806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.666859 13806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1016334e61d6f29b7ee02590f45bf25c"
server_key_iv: "3b890ca1a2b47ae1670d5620b5108ab6"
server_key_version: "encryptionkey@0"
I20260929 18:56:20.667229 13806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:20.674374 13806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:20.676312 13806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:20.676460 13806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:20.676681 13806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:20.676987 13806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:20.677028 13806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.677050 13806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:20.677086 13806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.683033 13806 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:32933
I20260929 18:56:20.683121 13934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:32933 every 8 connection(s)
I20260929 18:56:20.683460 13806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:56:20.685194 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13806
Sep 29 18:56:20 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:56:20.693451 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.685620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.29.71.190:45471)
0929 18:56:20.685712 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.685714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.685813 (+ 99us) server_negotiation.cc:407] Connection header received
0929 18:56:20.685885 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.685886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.685897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.685950 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:20.686717 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.687375 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.688497 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.688684 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.688763 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.690303 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:20.690304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:20.690304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:20.690322 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:20.692125 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.692549 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.692550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.692551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.692615 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.692863 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.692864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.692865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.693012 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:20.693192 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.693233 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.693305 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
I20260929 18:56:20.694628 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.684274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:38823 (local address 127.29.71.190:45471)
0929 18:56:20.684372 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.684375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.684470 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:56:20.685069 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.685070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.685080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.685123 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:20.686530 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.687285 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.688727 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.688934 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.688980 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.691608 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:20.691609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:20.691610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:20.691616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:20.693175 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.693640 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.693641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.693641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.693699 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.694064 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.694065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.694065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.694223 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:20.694289 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.694511 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.694571 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:56:20.694937 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.684207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:38823)
0929 18:56:20.684361 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.684400 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.684938 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.685211 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.685216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.686104 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:20.686439 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.686447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.687387 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.687388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.688659 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.688661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.689017 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.689018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.689144 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.689543 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:20.689554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:20.691521 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:20.693247 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:20.693253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:20.693256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:20.693544 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:20.693845 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:20.693846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:20.693846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:20.693999 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:20.694285 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:20.694290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:20.694410 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.694495 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.694614 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260929 18:56:20.695454 13935 heartbeater.cc:347] Connected to a master server at 127.29.71.190:45471
I20260929 18:56:20.695542 13935 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:20.695688 13935 heartbeater.cc:511] Master 127.29.71.190:45471 requested a full tablet report, sending...
I20260929 18:56:20.695986 13477 ts_manager.cc:194] Registered new tserver with Master: a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
I20260929 18:56:20.696408 13477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:38823
I20260929 18:56:20.705053 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:56:20.707793 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.705402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.29.71.190:45471)
0929 18:56:20.705465 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.705468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.705567 (+ 99us) server_negotiation.cc:407] Connection header received
0929 18:56:20.705623 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.705624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.705659 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.705723 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:56:20.705821 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.706389 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.707185 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.707360 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.707411 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.707585 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:56:20.707629 (+ 44us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 18:56:20.707746 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39458: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 18:56:20.707883 13668 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39458: token expired
W20260929 18:56:20.707984 13404 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.71.190:45471)
I20260929 18:56:20.714144 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.708244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.29.71.190:45471)
0929 18:56:20.708298 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.708300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.708307 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:56:20.708324 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.708325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.708332 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.708386 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:20.708818 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.709301 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.709981 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.710267 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.710321 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.711514 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:20.711515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:20.711516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:20.711522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:20.713034 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.713418 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.713419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.713419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.713473 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:20.713613 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:20.713614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:20.713614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:20.713743 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:20.713823 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.714021 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.714083 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:56:20.715289 13477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:56:20.716909 13477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:56:20.719259 13587 tablet_service.cc:1467] Processing CreateTablet for tablet e1fae2dd77514a26b0eb08a0ca2b1393 (DEFAULT_TABLE table=test-table [id=ab98f84e14fa4bc8b4e739e925c80e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:20.719466 13587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fae2dd77514a26b0eb08a0ca2b1393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:20.721047 13557 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap starting.
I20260929 18:56:20.721376 13557 tablet_bootstrap.cc:654] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:20.721938 13557 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: No bootstrap required, opened a new log
I20260929 18:56:20.721988 13557 ts_tablet_manager.cc:1397] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:20.722097 13557 raft_consensus.cc:348] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.722213 13557 raft_consensus.cc:374] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:20.722259 13557 raft_consensus.cc:729] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f4b26414d74bb09c54b43884d0ec3d, State: Initialized, Role: FOLLOWER
I20260929 18:56:20.722303 13557 consensus_queue.cc:260] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.722410 13557 ts_tablet_manager.cc:1428] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:20.722992 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.719038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.29.71.131:32933)
0929 18:56:20.719230 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.719236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.719247 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:56:20.719332 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.719334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.719350 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.719428 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:20.719555 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.720415 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.721719 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.722576 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.722643 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.722699 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.722861 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.722908 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260929 18:56:20.723013 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.718942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:32933 (local address 127.0.0.1:47192)
0929 18:56:20.719173 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.719185 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.719241 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.719431 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.719431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.719434 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:20.719486 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.719488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.720496 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.720497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.721583 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.721585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.722658 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.722659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.722765 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.722769 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.722836 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.722923 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":31,"thread_start_us":127,"threads_started":1}
I20260929 18:56:20.724253 13869 tablet_service.cc:1467] Processing CreateTablet for tablet e1fae2dd77514a26b0eb08a0ca2b1393 (DEFAULT_TABLE table=test-table [id=ab98f84e14fa4bc8b4e739e925c80e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:20.724556 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.718874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42549 (local address 127.0.0.1:49656)
0929 18:56:20.718931 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.718942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.718976 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.721404 (+ 2428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.721405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.721407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:20.721438 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.721439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.722772 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.722773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.723558 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.723560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.724321 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.724322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.724390 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.724394 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.724450 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.724494 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 18:56:20.724581 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.718999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.29.71.130:42549)
0929 18:56:20.721237 (+ 2238us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.721243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.721253 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:56:20.721290 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.721292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.721308 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.721496 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:20.721581 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.722674 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.723662 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.724237 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.724323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.724377 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.724465 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.724503 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2198,"thread_start_us":80,"threads_started":1}
I20260929 18:56:20.724582 13869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fae2dd77514a26b0eb08a0ca2b1393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:20.725188 13737 tablet_service.cc:1467] Processing CreateTablet for tablet e1fae2dd77514a26b0eb08a0ca2b1393 (DEFAULT_TABLE table=test-table [id=ab98f84e14fa4bc8b4e739e925c80e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:20.725466 13737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fae2dd77514a26b0eb08a0ca2b1393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:20.727569 13945 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Bootstrap starting.
I20260929 18:56:20.728302 13945 tablet_bootstrap.cc:654] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:20.728461 13946 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Bootstrap starting.
I20260929 18:56:20.728941 13945 log.cc:824] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:20.729221 13946 tablet_bootstrap.cc:654] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:20.729677 13945 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: No bootstrap required, opened a new log
I20260929 18:56:20.729758 13945 ts_tablet_manager.cc:1397] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:20.729769 13946 log.cc:824] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:20.730454 13946 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: No bootstrap required, opened a new log
I20260929 18:56:20.730507 13946 ts_tablet_manager.cc:1397] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:56:20.731340 13945 raft_consensus.cc:348] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.731505 13945 raft_consensus.cc:374] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:20.731551 13945 raft_consensus.cc:729] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a4715b594e4f4f92bf7ef5b1a0c817, State: Initialized, Role: FOLLOWER
I20260929 18:56:20.731670 13945 consensus_queue.cc:260] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.731931 13935 heartbeater.cc:503] Master 127.29.71.190:45471 was elected leader, sending a full tablet report...
I20260929 18:56:20.731938 13945 ts_tablet_manager.cc:1428] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:20.731902 13946 raft_consensus.cc:348] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.732023 13946 raft_consensus.cc:374] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:20.732043 13946 raft_consensus.cc:729] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f117256cfb4aee9f2be9c47f8166f3, State: Initialized, Role: FOLLOWER
I20260929 18:56:20.732142 13946 consensus_queue.cc:260] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.732399 13946 ts_tablet_manager.cc:1428] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:20.732515 13803 heartbeater.cc:503] Master 127.29.71.190:45471 was elected leader, sending a full tablet report...
I20260929 18:56:20.791585 13950 raft_consensus.cc:482] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:20.791715 13950 raft_consensus.cc:504] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.792145 13950 leader_election.cc:288] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933), 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413)
I20260929 18:56:20.795753 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.792424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:36481 (local address 127.29.71.129:37413)
0929 18:56:20.792473 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.792475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.792655 (+ 180us) server_negotiation.cc:407] Connection header received
0929 18:56:20.792708 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.792710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.792718 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.792756 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:20.792978 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.793662 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.794721 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.795288 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.795377 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.795386 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.795617 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.795664 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:56:20.795718 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.792349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:32933 (local address 127.29.71.130:41041)
0929 18:56:20.792396 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.792410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.792464 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.792647 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.792648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.792650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:20.792729 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.792730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.793702 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.793703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.794700 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.794703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.795454 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.795455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.795560 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.795563 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.795613 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.795650 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":10}
I20260929 18:56:20.795945 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.792468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:41041 (local address 127.29.71.131:32933)
0929 18:56:20.792523 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.792526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.792539 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:56:20.792565 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.792565 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.792573 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.792622 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:20.792841 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.793623 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.794813 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.795382 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.795450 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.795456 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.795819 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.795865 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:56:20.795733 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.792364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37413 (local address 127.29.71.130:36481)
0929 18:56:20.792573 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.792581 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:20.792648 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:20.792818 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:20.792819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:20.792821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:20.792893 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.792894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.793731 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.793733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.794630 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:20.794633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.795375 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.795376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:20.795456 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.795461 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:56:20.795534 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:56:20.795579 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":116,"threads_started":1}
I20260929 18:56:20.796216 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fae2dd77514a26b0eb08a0ca2b1393" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" is_pre_election: true
I20260929 18:56:20.796239 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fae2dd77514a26b0eb08a0ca2b1393" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f4b26414d74bb09c54b43884d0ec3d" is_pre_election: true
I20260929 18:56:20.796345 13889 raft_consensus.cc:2454] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 0.
I20260929 18:56:20.796363 13601 raft_consensus.cc:2454] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 0.
I20260929 18:56:20.796614 13692 leader_election.cc:302] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a4715b594e4f4f92bf7ef5b1a0c817, a9f117256cfb4aee9f2be9c47f8166f3; no voters:
I20260929 18:56:20.796761 13950 raft_consensus.cc:2781] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:56:20.796818 13950 raft_consensus.cc:482] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:56:20.796841 13950 raft_consensus.cc:3037] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:20.797559 13950 raft_consensus.cc:504] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.797709 13950 leader_election.cc:288] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 election: Requested vote from peers a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933), 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413)
I20260929 18:56:20.797879 13601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fae2dd77514a26b0eb08a0ca2b1393" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f4b26414d74bb09c54b43884d0ec3d"
I20260929 18:56:20.797967 13601 raft_consensus.cc:3037] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:20.799026 13601 raft_consensus.cc:2454] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 1.
I20260929 18:56:20.799153 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fae2dd77514a26b0eb08a0ca2b1393" candidate_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817"
I20260929 18:56:20.799206 13691 leader_election.cc:302] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d, a9f117256cfb4aee9f2be9c47f8166f3; no voters:
I20260929 18:56:20.799278 13889 raft_consensus.cc:3037] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:20.799317 13950 raft_consensus.cc:2781] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:56:20.799454 13950 raft_consensus.cc:686] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 LEADER]: Becoming Leader. State: Replica: a9f117256cfb4aee9f2be9c47f8166f3, State: Running, Role: LEADER
I20260929 18:56:20.799551 13950 consensus_queue.cc:237] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:20.800122 13889 raft_consensus.cc:2454] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f117256cfb4aee9f2be9c47f8166f3 in term 1.
I20260929 18:56:20.800503 13476 catalog_manager.cc:5495] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130). New cstate: current_term: 1 leader_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } health_report { overall_health: UNKNOWN } } }
W20260929 18:56:20.811645 13804 tablet.cc:2369] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:56:20.816314 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.814065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.29.71.130:42549)
0929 18:56:20.814120 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:20.814122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:20.814131 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:56:20.814248 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:20.814249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:20.814266 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:20.814336 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:56:20.814445 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.814946 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.815636 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:20.815845 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:20.815895 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:20.816022 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:56:20.816122 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:56:20.816182 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:20.816219 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 18:56:20.816237 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:56:20.818205 13889 raft_consensus.cc:1261] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Refusing update from remote peer a9f117256cfb4aee9f2be9c47f8166f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:20.818303 13601 raft_consensus.cc:1261] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Refusing update from remote peer a9f117256cfb4aee9f2be9c47f8166f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:20.818579 13950 consensus_queue.cc:1035] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:20.818758 13952 consensus_queue.cc:1035] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:20.823529 13476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:20.823635 13476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:20.824357 13476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Sending DeleteTablet for 3 replicas of tablet e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:20.824672 13869 tablet_service.cc:1514] Processing DeleteTablet for tablet e1fae2dd77514a26b0eb08a0ca2b1393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:20 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:47192
I20260929 18:56:20.824680 13737 tablet_service.cc:1514] Processing DeleteTablet for tablet e1fae2dd77514a26b0eb08a0ca2b1393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:20 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:49656
I20260929 18:56:20.824708 13587 tablet_service.cc:1514] Processing DeleteTablet for tablet e1fae2dd77514a26b0eb08a0ca2b1393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:20 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:33260
I20260929 18:56:20.824791 13674 tablet_replica.cc:330] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: stopping tablet replica
I20260929 18:56:20.824842 13674 raft_consensus.cc:2229] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:56:20.824903 13674 raft_consensus.cc:2258] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:20.824949 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 13515
I20260929 18:56:20.825103 13963 tablet_replica.cc:330] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: stopping tablet replica
I20260929 18:56:20.825204 13963 raft_consensus.cc:2229] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:56:20.825304 13963 raft_consensus.cc:2258] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:20.825731 13963 ts_tablet_manager.cc:1905] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:56:20.826274 13964 tablet_replica.cc:330] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: stopping tablet replica
I20260929 18:56:20.826381 13964 raft_consensus.cc:2229] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:56:20.826475 13964 raft_consensus.cc:2258] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:20.828048 13963 ts_tablet_manager.cc:1918] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:20.828120 13963 log.cc:1196] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:20.828416 13963 ts_tablet_manager.cc:1939] T e1fae2dd77514a26b0eb08a0ca2b1393 P a4a4715b594e4f4f92bf7ef5b1a0c817: Deleting consensus metadata
I20260929 18:56:20.828908 13463 catalog_manager.cc:4841] TS a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933): tablet e1fae2dd77514a26b0eb08a0ca2b1393 (table test-table [id=ab98f84e14fa4bc8b4e739e925c80e7d]) successfully deleted
I20260929 18:56:20.830509 13964 ts_tablet_manager.cc:1905] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:56:20.831840 13462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:37413 (error 108)
I20260929 18:56:20.832338 13964 ts_tablet_manager.cc:1918] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:20.832413 13964 log.cc:1196] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:20.832491 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 13670
I20260929 18:56:20.833833 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.833689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37413 (local address 127.0.0.1:33272)
0929 18:56:20.833745 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.833801 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 18:56:20.833961 13462 catalog_manager.cc:4584] TS 66f4b26414d74bb09c54b43884d0ec3d (127.29.71.129:37413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1fae2dd77514a26b0eb08a0ca2b1393: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37413: connect: Connection refused (error 111)
I20260929 18:56:20.837415 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 13806
I20260929 18:56:20.837448 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:20.837345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42549 (local address 127.0.0.1:49684)
0929 18:56:20.837396 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:20.837424 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.130:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 18:56:20.842020 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 13444
I20260929 18:56:40.847740 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45471
--webserver_interface=127.29.71.190
--webserver_port=45899
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:56:40.929656 13966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:40.929924 13966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:40.929960 13966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:40.931196 13966 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:56:40.931272 13966 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:56:40.931293 13966 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:56:40.931313 13966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:56:40.931340 13966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:40.931372 13966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:40.931401 13966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:56:40.931429 13966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:56:40.933270 13966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:45471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=45899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:40.933609 13966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:40.933874 13966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:56:40.939381 13966 server_base.cc:1034] running on GCE node
W20260929 18:56:40.939381 13971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:40.939405 13972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:40.939546 13974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:40.939847 13966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:40.940104 13966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:40.941285 13966 hybrid_clock.cc:648] HybridClock initialized: now 1790708200941262 us; error 41 us; skew 500 ppm
Sep 29 18:56:40 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708200, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:40.947925 13966 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:56:40.948477 13966 webserver.cc:460] Webserver started at http://127.29.71.190:45899/ using document root <none> and password file <none>
I20260929 18:56:40.948724 13966 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:40.948796 13966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:40.950297 13966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:40.951025 13981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:40.951265 13966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:40.951378 13966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "9a253abdb3394bb8a536ea903eee3690"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "3d97d615713ebf66295f6a2b27cc80eb"
server_key_iv: "d5de643d61ab2f8fb8c59c1c30850503"
server_key_version: "encryptionkey@0"
I20260929 18:56:40.951764 13966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:40.973454 13966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:40.976229 13966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:40.976413 13966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:40.980743 13966 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:45471
I20260929 18:56:40.980772 14033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:45471 every 8 connection(s)
I20260929 18:56:40.981159 13966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:56:40.982738 14034 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:56:40.983148 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 13966
I20260929 18:56:40.983510 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:37413
--local_ip_for_outbound_sockets=127.29.71.129
--tserver_master_addrs=127.29.71.190:45471
--webserver_port=34047
--webserver_interface=127.29.71.129
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:56:40.983781 14034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Bootstrap starting.
I20260929 18:56:40.993139 14034 log.cc:824] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:40.995756 14034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:56:40.996099 14034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Bootstrap complete.
I20260929 18:56:40.998767 14034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:40.999032 14034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a253abdb3394bb8a536ea903eee3690, State: Initialized, Role: FOLLOWER
I20260929 18:56:40.999217 14034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:40.999289 14034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:56:40.999346 14034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:56:40.999395 14034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:56:41.000516 14034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:41.000614 14034 leader_election.cc:302] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a253abdb3394bb8a536ea903eee3690; no voters:
I20260929 18:56:41.000864 14034 leader_election.cc:288] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:56:41.000901 14038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:56:41.001030 14038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [term 3 LEADER]: Becoming Leader. State: Replica: 9a253abdb3394bb8a536ea903eee3690, State: Running, Role: LEADER
I20260929 18:56:41.001132 14038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } }
I20260929 18:56:41.001178 14034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:56:41.001441 14040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a253abdb3394bb8a536ea903eee3690. Latest consensus state: current_term: 3 leader_uuid: "9a253abdb3394bb8a536ea903eee3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } } }
I20260929 18:56:41.001660 14040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:41.001605 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9a253abdb3394bb8a536ea903eee3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a253abdb3394bb8a536ea903eee3690" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 45471 } } }
I20260929 18:56:41.001919 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:41.002059 14047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:56:41.002894 14047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:56:41.003010 14047 catalog_manager.cc:1259] Loaded cluster ID: d5e04108f21448f38be0da1bc4c0c091
I20260929 18:56:41.003271 14047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:56:41.004437 14047 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:56:41.004616 14047 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Loaded TSK: 0
I20260929 18:56:41.004659 14047 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a253abdb3394bb8a536ea903eee3690: Loaded TSK: 1
F20260929 18:56:41.004916 14047 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd17abd89ad google::LogMessage::Fail() at ??:0
@ 0x7fd17abdb88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd17abd84de google::LogMessage::Flush() at ??:0
@ 0x7fd17abd9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd17d345104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd17d342d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd182b321cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd182b3248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd182b32594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd182af8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd182b02ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd17caa60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd17ca9b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd17d8ff6db start_thread at ??:0
@ 0x7fd1796d661f clone at ??:0
W20260929 18:56:41.089753 14036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:41.090010 14036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:41.090054 14036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:41.091382 14036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:41.091454 14036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:41.091506 14036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:56:41.093097 14036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:37413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=34047
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:41.093384 14036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:41.093631 14036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:41.099206 14076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:41.099232 14075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:41.099255 14078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:41.099512 14036 server_base.cc:1034] running on GCE node
I20260929 18:56:41.099721 14036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:41.100033 14036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:41.101199 14036 hybrid_clock.cc:648] HybridClock initialized: now 1790708201101179 us; error 37 us; skew 500 ppm
Sep 29 18:56:41 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708201, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:41.104009 14036 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:56:41.104465 14036 webserver.cc:460] Webserver started at http://127.29.71.129:34047/ using document root <none> and password file <none>
I20260929 18:56:41.104710 14036 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:41.104787 14036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:41.106312 14036 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:41.107136 14085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:41.107348 14036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:41.107467 14036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "66f4b26414d74bb09c54b43884d0ec3d"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "0b09637df0738c2e9756355cc75f14c2"
server_key_iv: "a2d5f0e48b835ef27c8b7a826845d0ed"
server_key_version: "encryptionkey@0"
I20260929 18:56:41.107849 14036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:41.121814 14036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:41.124351 14036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:41.124536 14036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:41.124788 14036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:41.125252 14092 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:56:41.127213 14036 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:56:41.127290 14036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 18:56:41.127339 14036 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:56:41.128046 14036 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:56:41.128113 14036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 18:56:41.128173 14092 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap starting.
I20260929 18:56:41.136082 14036 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37413
I20260929 18:56:41.136227 14199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37413 every 8 connection(s)
I20260929 18:56:41.136569 14036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:56:41.139142 14092 log.cc:824] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:41.139475 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 14036
I20260929 18:56:41.139799 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:42549
--local_ip_for_outbound_sockets=127.29.71.130
--tserver_master_addrs=127.29.71.190:45471
--webserver_port=46065
--webserver_interface=127.29.71.130
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:56:41.141791 14092 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:56:41.142132 14092 tablet_bootstrap.cc:492] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Bootstrap complete.
I20260929 18:56:41.143447 14092 ts_tablet_manager.cc:1397] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260929 18:56:41.144857 14092 raft_consensus.cc:348] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:41.145177 14092 raft_consensus.cc:729] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f4b26414d74bb09c54b43884d0ec3d, State: Initialized, Role: FOLLOWER
I20260929 18:56:41.145342 14092 consensus_queue.cc:260] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:41.145619 14092 ts_tablet_manager.cc:1428] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260929 18:56:41.211746 14202 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:56:41.137294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:50379)
0929 18:56:41.137470 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.137516 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:41.138021 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:41.211432 (+ 73411us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
W20260929 18:56:41.212066 14200 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:45471 (0 consecutive failures): Network error: Failed to ping master at 127.29.71.190:45471: Client connection negotiation failed: client connection to 127.29.71.190:45471: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:56:41.212461 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.212321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:51155)
0929 18:56:41.212383 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.212435 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:56:41.212773 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.212699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:39769)
0929 18:56:41.212727 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.212754 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 18:56:41.212946 14200 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:56:41.229202 14204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:41.229434 14204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:41.229475 14204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:41.230738 14204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:41.230801 14204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:41.230850 14204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:56:41.233210 14204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=46065
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:41.233450 14204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:41.233721 14204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:41.240935 14210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:41.240993 14204 server_base.cc:1034] running on GCE node
W20260929 18:56:41.241134 14211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:41.240959 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:41.241431 14204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:41.241719 14204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:41.242848 14204 hybrid_clock.cc:648] HybridClock initialized: now 1790708201242833 us; error 30 us; skew 500 ppm
Sep 29 18:56:41 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708201, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:41.246505 14204 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:56:41.247048 14204 webserver.cc:460] Webserver started at http://127.29.71.130:46065/ using document root <none> and password file <none>
I20260929 18:56:41.247313 14204 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:41.247392 14204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:41.249171 14204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:41.249977 14220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:41.250135 14204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:41.250263 14204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "a9f117256cfb4aee9f2be9c47f8166f3"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1cc3e846bfbdb5969641cca55bddf1e6"
server_key_iv: "ec58daba3a9001e864b7651a10b27cf0"
server_key_version: "encryptionkey@0"
I20260929 18:56:41.250787 14204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:41.289433 14204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:41.291745 14204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:41.291920 14204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:41.292196 14204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:41.292676 14227 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 18:56:41.294139 14227 ts_tablet_manager.cc:1852] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 18:56:41.294502 14227 ts_tablet_manager.cc:1905] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:56:41.296670 14227 ts_tablet_manager.cc:1918] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:41.296828 14227 log.cc:1196] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:41.297070 14227 ts_tablet_manager.cc:1939] T e1fae2dd77514a26b0eb08a0ca2b1393 P a9f117256cfb4aee9f2be9c47f8166f3: Deleting consensus metadata
I20260929 18:56:41.297298 14204 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 18:56:41.297349 14204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 18:56:41.297376 14204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:41.297389 14204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:41.303531 14204 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:42549
I20260929 18:56:41.303583 14334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:42549 every 8 connection(s)
I20260929 18:56:41.303882 14204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:56:41.304932 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.304523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:36463)
0929 18:56:41.304678 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.304784 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
W20260929 18:56:41.305213 14335 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:45471 (0 consecutive failures): Network error: Failed to ping master at 127.29.71.190:45471: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
I20260929 18:56:41.305562 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.305417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:33215)
0929 18:56:41.305493 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.305530 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 18:56:41.305645 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 14204
I20260929 18:56:41.305938 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:32933
--local_ip_for_outbound_sockets=127.29.71.131
--tserver_master_addrs=127.29.71.190:45471
--webserver_port=36611
--webserver_interface=127.29.71.131
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:56:41.305960 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.305851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:58787)
0929 18:56:41.305904 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.305935 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:56:41.306135 14335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:56:41.386427 14338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:41.386627 14338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:41.386649 14338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:41.387755 14338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:41.387825 14338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:41.387871 14338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:56:41.389287 14338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:32933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=36611
--tserver_master_addrs=127.29.71.190:45471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:41.389565 14338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:41.389799 14338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:41.395076 14344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:41.395002 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:41.395078 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:41.395244 14338 server_base.cc:1034] running on GCE node
I20260929 18:56:41.395433 14338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:41.395692 14338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:41.396862 14338 hybrid_clock.cc:648] HybridClock initialized: now 1790708201396862 us; error 42 us; skew 500 ppm
Sep 29 18:56:41 dist-test-slave-q2kh krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708201, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:41.399453 14338 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:56:41.399940 14338 webserver.cc:460] Webserver started at http://127.29.71.131:36611/ using document root <none> and password file <none>
I20260929 18:56:41.400188 14338 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:41.400259 14338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:41.401602 14338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:41.402258 14353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:41.402432 14338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:41.402529 14338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817"
format_stamp: "Formatted at 2026-09-29 18:55:59 on dist-test-slave-q2kh"
server_key: "1016334e61d6f29b7ee02590f45bf25c"
server_key_iv: "3b890ca1a2b47ae1670d5620b5108ab6"
server_key_version: "encryptionkey@0"
I20260929 18:56:41.403020 14338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:41.414525 14338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:41.416900 14338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:41.417068 14338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:41.417316 14338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:41.417632 14338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:41.417685 14338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:41.417721 14338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:41.417747 14338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:41.423581 14338 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:32933
I20260929 18:56:41.423632 14466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:32933 every 8 connection(s)
I20260929 18:56:41.423961 14338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:56:41.424973 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.424599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:51669)
0929 18:56:41.424739 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.424808 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260929 18:56:41.425391 14467 heartbeater.cc:644] Failed to heartbeat to 127.29.71.190:45471 (0 consecutive failures): Network error: Failed to ping master at 127.29.71.190:45471: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
I20260929 18:56:41.425684 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.425599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:37329)
0929 18:56:41.425642 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.425668 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:56:41.425942 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:41.425891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:57787)
0929 18:56:41.425911 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:41.425930 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260929 18:56:41.426061 14467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:56:41.430873 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 14338
I20260929 18:56:42.213591 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:42.213340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:32789)
0929 18:56:42.213481 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:42.213527 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260929 18:56:42.306694 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:42.306486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:35689)
0929 18:56:42.306601 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:42.306641 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260929 18:56:42.426568 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:42.426379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:37045)
0929 18:56:42.426487 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:42.426517 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260929 18:56:42.849314 14474 raft_consensus.cc:482] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:42.849488 14474 raft_consensus.cc:504] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:42.849861 14474 leader_election.cc:288] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549), a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
Sep 29 18:56:42 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708201, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.130@KRBTEST.COM
Sep 29 18:56:42 dist-test-slave-q2kh krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708201, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.131@KRBTEST.COM
I20260929 18:56:42.861334 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:42.850189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:56953 (local address 127.29.71.131:32933)
0929 18:56:42.850335 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:42.850347 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:42.850380 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:56:42.850964 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:42.850967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:42.851037 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:42.851177 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:42.851980 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.852915 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.853983 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.854549 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.854773 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:42.857773 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:42.857776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:42.857777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:42.857790 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:42.859650 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:42.860090 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:42.860093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:42.860093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:42.860217 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:42.860457 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:42.860457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:42.860458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:42.860702 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:42.860799 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:42.861087 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 18:56:42.861212 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260929 18:56:42.861320 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:42.849987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:42549 (local address 127.29.71.129:41719)
0929 18:56:42.850177 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:42.850189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:42.850251 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:42.851197 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:42.851199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:42.851562 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:42.851833 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:42.851839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.852964 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.852965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:42.853824 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:42.853826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.854615 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.854616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:42.854728 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:42.855430 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:42.855434 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:42.857338 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:42.859488 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:42.859496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:42.859498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:42.859875 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:42.860193 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:42.860195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:42.860195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:42.860328 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:42.860770 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:42.860777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:42.860907 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 18:56:42.861026 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 18:56:42.861225 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":33,"thread_start_us":111,"threads_started":1}
I20260929 18:56:42.861402 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:42.850129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:41719 (local address 127.29.71.130:42549)
0929 18:56:42.850381 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:42.850390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:42.850410 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:56:42.850932 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:42.850935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:42.850995 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:42.851155 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:42.851956 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.852834 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.853945 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.854507 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.854642 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:42.857467 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:42.857471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:42.857472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:42.857485 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:42.859390 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:42.859978 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:42.859982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:42.859983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:42.860108 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:42.860410 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:42.860411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:42.860411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:42.860694 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:42.860792 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:42.861090 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 18:56:42.861259 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
W20260929 18:56:42.862099 14086 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:42.862116 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:42.850027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:32933 (local address 127.29.71.129:56953)
0929 18:56:42.850200 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:42.850211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:42.850275 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:42.851211 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:42.851212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:42.851657 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:42.851834 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:42.851840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.853022 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.853023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:42.853877 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:42.853880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:42.854653 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:42.854654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:42.854728 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:42.855406 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:42.855418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:42.857664 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:42.859753 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:42.859753 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:42.859754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:42.860002 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:42.860293 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:42.860294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:42.860294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:42.860356 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:42.860801 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:42.860802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:42.860907 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 18:56:42.861027 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:56:42.861317 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":160,"thread_start_us":91,"threads_started":1}
W20260929 18:56:42.862891 14088 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:42.862953 14088 leader_election.cc:302] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d; no voters: a4a4715b594e4f4f92bf7ef5b1a0c817, a9f117256cfb4aee9f2be9c47f8166f3
I20260929 18:56:42.863075 14474 raft_consensus.cc:2726] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:56:43.214449 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:43.214287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:41773)
0929 18:56:43.214348 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:43.214409 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:56:43.307408 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:43.307198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:60873)
0929 18:56:43.307332 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:43.307367 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260929 18:56:43.427205 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:43.426982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:55559)
0929 18:56:43.427125 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:43.427166 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260929 18:56:44.215231 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:44.214967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:41149)
0929 18:56:44.215121 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:44.215170 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260929 18:56:44.308096 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:44.307882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:35479)
0929 18:56:44.308001 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:44.308042 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260929 18:56:44.427896 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:44.427671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:58609)
0929 18:56:44.427812 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:44.427855 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260929 18:56:44.515977 14484 raft_consensus.cc:482] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:44.516080 14484 raft_consensus.cc:504] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:44.516302 14484 leader_election.cc:288] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549), a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
W20260929 18:56:44.516798 14086 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
W20260929 18:56:44.516950 14088 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:44.516987 14088 leader_election.cc:302] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d; no voters: a4a4715b594e4f4f92bf7ef5b1a0c817, a9f117256cfb4aee9f2be9c47f8166f3
I20260929 18:56:44.517099 14484 raft_consensus.cc:2726] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:56:45.215931 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:45.215726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:50559)
0929 18:56:45.215849 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:45.215883 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260929 18:56:45.308750 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:45.308549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:47189)
0929 18:56:45.308665 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:45.308708 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260929 18:56:45.428601 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:45.428408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:54103)
0929 18:56:45.428528 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:45.428563 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260929 18:56:46.216740 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:46.216475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:33379)
0929 18:56:46.216624 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:46.216681 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260929 18:56:46.309506 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:46.309265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:39577)
0929 18:56:46.309395 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:46.309462 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260929 18:56:46.429514 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:46.429303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:56091)
0929 18:56:46.429435 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:46.429471 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260929 18:56:46.537845 14491 raft_consensus.cc:482] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:46.537967 14491 raft_consensus.cc:504] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:46.538136 14491 leader_election.cc:288] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549), a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
W20260929 18:56:46.538575 14088 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
W20260929 18:56:46.538641 14086 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:46.538671 14088 leader_election.cc:302] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d; no voters: a4a4715b594e4f4f92bf7ef5b1a0c817, a9f117256cfb4aee9f2be9c47f8166f3
I20260929 18:56:46.538741 14491 raft_consensus.cc:2726] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:56:47.217603 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:47.217315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:55773)
0929 18:56:47.217476 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:47.217522 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260929 18:56:47.310410 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:47.310134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:59797)
0929 18:56:47.310285 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:47.310343 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260929 18:56:47.430359 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:47.430074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:60383)
0929 18:56:47.430271 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:47.430315 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260929 18:56:48.218485 14495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:48.218272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:43603)
0929 18:56:48.218397 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:48.218437 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260929 18:56:48.311385 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:48.311081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:42141)
0929 18:56:48.311297 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:48.311336 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":138,"threads_started":1}
I20260929 18:56:48.431100 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:48.430855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:50925)
0929 18:56:48.431013 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:48.431056 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260929 18:56:49.219165 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:49.218933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:41765)
0929 18:56:49.219082 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:49.219119 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260929 18:56:49.285729 14499 raft_consensus.cc:482] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:49.285845 14499 raft_consensus.cc:504] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:49.286075 14499 leader_election.cc:288] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549), a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
W20260929 18:56:49.286514 14088 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
W20260929 18:56:49.286629 14086 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:49.286665 14086 leader_election.cc:302] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d; no voters: a4a4715b594e4f4f92bf7ef5b1a0c817, a9f117256cfb4aee9f2be9c47f8166f3
I20260929 18:56:49.286785 14499 raft_consensus.cc:2726] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:56:49.312198 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:49.311950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:58735)
0929 18:56:49.312108 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:49.312151 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260929 18:56:49.431795 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:49.431592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:45347)
0929 18:56:49.431713 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:49.431749 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260929 18:56:50.219830 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:50.219614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:47305)
0929 18:56:50.219742 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:50.219782 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260929 18:56:50.312915 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:50.312679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:41067)
0929 18:56:50.312827 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:50.312869 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260929 18:56:50.432497 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:50.432288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:54123)
0929 18:56:50.432415 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:50.432447 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260929 18:56:51.220650 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:51.220382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:48567)
0929 18:56:51.220558 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:51.220601 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260929 18:56:51.313679 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:51.313472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:38149)
0929 18:56:51.313588 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:51.313624 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260929 18:56:51.433192 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:51.432975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:38233)
0929 18:56:51.433116 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:51.433148 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260929 18:56:52.221408 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:52.221115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:33347)
0929 18:56:52.221300 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:52.221346 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260929 18:56:52.314407 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:52.314151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:48385)
0929 18:56:52.314302 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:52.314349 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
I20260929 18:56:52.433874 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:52.433640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:33641)
0929 18:56:52.433790 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:52.433828 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260929 18:56:53.222149 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:53.221917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:45421)
0929 18:56:53.222062 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:53.222103 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260929 18:56:53.315205 14512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:53.314948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:50279)
0929 18:56:53.315116 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:53.315158 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260929 18:56:53.434633 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:53.434413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:56023)
0929 18:56:53.434547 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:53.434588 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260929 18:56:54.222951 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:54.222721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:39497)
0929 18:56:54.222862 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:54.222904 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260929 18:56:54.315905 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:54.315690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:46441)
0929 18:56:54.315819 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:54.315852 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260929 18:56:54.435355 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:54.435144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:51409)
0929 18:56:54.435273 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:54.435310 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260929 18:56:54.896783 14517 raft_consensus.cc:482] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:54.896914 14517 raft_consensus.cc:504] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f117256cfb4aee9f2be9c47f8166f3" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 42549 } } peers { permanent_uuid: "a4a4715b594e4f4f92bf7ef5b1a0c817" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 32933 } } peers { permanent_uuid: "66f4b26414d74bb09c54b43884d0ec3d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37413 } }
I20260929 18:56:54.897128 14517 leader_election.cc:288] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549), a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933)
W20260929 18:56:54.897526 14088 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4a4715b594e4f4f92bf7ef5b1a0c817 (127.29.71.131:32933): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
W20260929 18:56:54.897600 14086 leader_election.cc:341] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f117256cfb4aee9f2be9c47f8166f3 (127.29.71.130:42549): Not found: Tablet not found: e1fae2dd77514a26b0eb08a0ca2b1393
I20260929 18:56:54.897675 14086 leader_election.cc:302] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66f4b26414d74bb09c54b43884d0ec3d; no voters: a4a4715b594e4f4f92bf7ef5b1a0c817, a9f117256cfb4aee9f2be9c47f8166f3
I20260929 18:56:54.897751 14517 raft_consensus.cc:2726] T e1fae2dd77514a26b0eb08a0ca2b1393 P 66f4b26414d74bb09c54b43884d0ec3d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:56:55.223614 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:55.223415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:41563)
0929 18:56:55.223534 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:55.223570 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260929 18:56:55.316574 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:55.316334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:41497)
0929 18:56:55.316498 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:55.316530 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260929 18:56:55.436110 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:55.435830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:33913)
0929 18:56:55.436011 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:55.436061 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":63,"threads_started":1}
I20260929 18:56:56.224481 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:56.224238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.129:52365)
0929 18:56:56.224385 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:56.224435 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260929 18:56:56.317401 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:56.317129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.130:48747)
0929 18:56:56.317277 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:56.317330 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260929 18:56:56.437091 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:56.436853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:45471 (local address 127.29.71.131:40535)
0929 18:56:56.436999 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:56.437046 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.190:45471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260929 18:56:56.441759 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 14036
I20260929 18:56:56.446362 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 14204
I20260929 18:56:56.450877 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 14338
2026-09-29T18:56:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58706 ms)
[----------] 30 tests from SecurityITest (261943 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:56:56 dist-test-slave-q2kh krb5kdc[14526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:56:56 dist-test-slave-q2kh krb5kdc[14526](info): set up 2 sockets
Sep 29 18:56:56 dist-test-slave-q2kh krb5kdc[14526](info): commencing operation
krb5kdc: starting...
W20260929 18:56:57.817785 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:56:57 dist-test-slave-q2kh krb5kdc[14526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:56:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:56:57.938350 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:39965
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:39965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:56:58.021370 14539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:58.021587 14539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:58.021611 14539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:58.022866 14539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:56:58.022914 14539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:58.022930 14539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:58.022945 14539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:56:58.023000 14539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:56:58.024479 14539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:39965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:39965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:58.024821 14539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:58.025091 14539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:58.030303 14545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:58.030319 14547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:58.030378 14539 server_base.cc:1034] running on GCE node
W20260929 18:56:58.030319 14544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:58.030843 14539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:58.031100 14539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:58.032253 14539 hybrid_clock.cc:648] HybridClock initialized: now 1790708218032239 us; error 27 us; skew 500 ppm
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708218, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:58.034888 14539 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:56:58.035378 14539 webserver.cc:460] Webserver started at http://127.29.71.190:35991/ using document root <none> and password file <none>
I20260929 18:56:58.035619 14539 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:58.035697 14539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:58.035856 14539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:58.036954 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "07ea429d67f840dc9ed5087b058636f3"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "d348fec685d1e02e5f560ff08aa27a0c"
server_key_iv: "ffa8a9a5cbc445842c6029edfdd8644b"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.037339 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "07ea429d67f840dc9ed5087b058636f3"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "d348fec685d1e02e5f560ff08aa27a0c"
server_key_iv: "ffa8a9a5cbc445842c6029edfdd8644b"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.038836 14539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:56:58.039615 14554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.039798 14539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:56:58.039902 14539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "07ea429d67f840dc9ed5087b058636f3"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "d348fec685d1e02e5f560ff08aa27a0c"
server_key_iv: "ffa8a9a5cbc445842c6029edfdd8644b"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.039991 14539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:58.051065 14539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:58.053545 14539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:58.053725 14539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:58.058065 14539 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:39965
I20260929 18:56:58.058099 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:39965 every 8 connection(s)
I20260929 18:56:58.058496 14539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:56:58.059340 14607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:58.061496 14607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3: Bootstrap starting.
I20260929 18:56:58.062004 14607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:58.062314 14607 log.cc:824] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:58.063004 14607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3: No bootstrap required, opened a new log
I20260929 18:56:58.063236 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 14539
I20260929 18:56:58.063349 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:56:58.063532 29982 external_mini_cluster.cc:1428] Setting key f962d4ecaffbca04757c25daa0885026
I20260929 18:56:58.065249 14607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ea429d67f840dc9ed5087b058636f3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39965 } }
I20260929 18:56:58.065435 14607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:58.065474 14607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ea429d67f840dc9ed5087b058636f3, State: Initialized, Role: FOLLOWER
I20260929 18:56:58.065631 14607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ea429d67f840dc9ed5087b058636f3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39965 } }
I20260929 18:56:58.065698 14607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:56:58.065719 14607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:56:58.065788 14607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:58.066855 14607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ea429d67f840dc9ed5087b058636f3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39965 } }
I20260929 18:56:58.066967 14607 leader_election.cc:302] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07ea429d67f840dc9ed5087b058636f3; no voters:
I20260929 18:56:58.067118 14607 leader_election.cc:288] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:56:58.067173 14612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:56:58.067358 14612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [term 1 LEADER]: Becoming Leader. State: Replica: 07ea429d67f840dc9ed5087b058636f3, State: Running, Role: LEADER
I20260929 18:56:58.067392 14607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:56:58.067507 14612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ea429d67f840dc9ed5087b058636f3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39965 } }
I20260929 18:56:58.067830 14614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07ea429d67f840dc9ed5087b058636f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ea429d67f840dc9ed5087b058636f3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39965 } } }
I20260929 18:56:58.067816 14612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07ea429d67f840dc9ed5087b058636f3. Latest consensus state: current_term: 1 leader_uuid: "07ea429d67f840dc9ed5087b058636f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ea429d67f840dc9ed5087b058636f3" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39965 } } }
I20260929 18:56:58.067909 14614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:58.067919 14612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:58.068274 14618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:56:58.068835 14618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:56:58.070513 14618 catalog_manager.cc:1347] Generated new cluster ID: a44afd3098604fd9860152fc4c0640e3
I20260929 18:56:58.070565 14618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:56:58.076223 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.064027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.29.71.190:39965)
0929 18:56:58.064215 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.064225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.064280 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:56:58.065212 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.065221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.065289 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.065438 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:58.066324 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.067720 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.068924 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.069588 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.069694 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.072588 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:58.072592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:58.072592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:58.072604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:58.074534 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.074936 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.074938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.074939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.075060 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.075288 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.075288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.075289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.075574 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:58.075715 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.075895 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.076028 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260929 18:56:58.080073 14618 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:56:58.080591 14618 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:56:58.086027 14618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3: Generated new TSK 0
I20260929 18:56:58.086215 14618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:56:58.121553 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39965
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:56:58.207620 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:58.207865 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:58.207906 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:58.209117 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:58.209185 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:58.209234 14633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:56:58.210796 14633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:58.211073 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:58.211302 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:58.216403 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:58.216547 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:58.216616 14633 server_base.cc:1034] running on GCE node
W20260929 18:56:58.216737 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:58.216936 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:58.217190 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:58.218346 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790708218218342 us; error 37 us; skew 500 ppm
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708218, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:58.220777 14633 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:56:58.221233 14633 webserver.cc:460] Webserver started at http://127.29.71.129:40193/ using document root <none> and password file <none>
I20260929 18:56:58.221468 14633 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:58.221536 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:58.221679 14633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:58.222805 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "4bf3d74c9ba78ec6a4e2fd789d942a66"
server_key_iv: "cd5c95228a19a24b42a29c898b533921"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.223161 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "4bf3d74c9ba78ec6a4e2fd789d942a66"
server_key_iv: "cd5c95228a19a24b42a29c898b533921"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.224624 14633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:58.225584 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.225896 14633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:58.225996 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "4bf3d74c9ba78ec6a4e2fd789d942a66"
server_key_iv: "cd5c95228a19a24b42a29c898b533921"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.226074 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:58.237040 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:58.239193 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:58.239351 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:58.239596 14633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:58.239900 14633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:58.239953 14633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.239995 14633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:58.240020 14633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.246294 14633 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37501
I20260929 18:56:58.246335 14761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37501 every 8 connection(s)
I20260929 18:56:58.246670 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:56:58.246963 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 14633
I20260929 18:56:58.247074 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:56:58.247282 29982 external_mini_cluster.cc:1428] Setting key 61d9fd66b18da4ec8ec8d752b7be004c
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708218, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
I20260929 18:56:58.256907 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.247444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:47711 (local address 127.29.71.190:39965)
0929 18:56:58.247491 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.247493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.247761 (+ 268us) server_negotiation.cc:407] Connection header received
0929 18:56:58.248334 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.248335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.248352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.248429 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:58.249614 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.250099 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.251150 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.251538 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.251616 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.253861 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:58.253863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:58.253863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:58.253876 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:58.255602 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.255990 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.255991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.255991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.256045 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.256267 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.256267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.256268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.256428 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:58.256496 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.256765 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.256838 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:56:58.257093 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.247451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:39965 (local address 127.29.71.129:47711)
0929 18:56:58.247626 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.247668 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.248234 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.248489 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.248493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.249213 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:58.249510 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.249517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.250245 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.250246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.251061 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.251063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.251627 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.251628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.251737 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.252340 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:58.252348 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:58.253763 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:58.255679 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:58.255686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:58.255687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:58.255918 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:58.256105 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:58.256106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:58.256106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:58.256208 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:58.256508 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:58.256515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:58.256645 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.256735 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.256858 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260929 18:56:58.257648 14762 heartbeater.cc:347] Connected to a master server at 127.29.71.190:39965
I20260929 18:56:58.257750 14762 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:58.257954 14762 heartbeater.cc:511] Master 127.29.71.190:39965 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
I20260929 18:56:58.258562 14571 ts_manager.cc:194] Registered new tserver with Master: e9b6c8ebf2e040c3b7d17389a26fdc6d (127.29.71.129:37501)
I20260929 18:56:58.259181 14571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:47711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:56:58.275270 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39965
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:56:58.355261 14767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:58.355510 14767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:58.355549 14767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:58.356748 14767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:58.356819 14767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:58.356869 14767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:56:58.358400 14767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:58.358680 14767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:58.358917 14767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:58.364245 14772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:58.364208 14775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:58.364229 14773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:58.364491 14767 server_base.cc:1034] running on GCE node
I20260929 18:56:58.364683 14767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:58.364895 14767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:58.366062 14767 hybrid_clock.cc:648] HybridClock initialized: now 1790708218366045 us; error 30 us; skew 500 ppm
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708218, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:58.368888 14767 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:56:58.369344 14767 webserver.cc:460] Webserver started at http://127.29.71.130:40935/ using document root <none> and password file <none>
I20260929 18:56:58.369591 14767 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:58.369663 14767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:58.369807 14767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:58.370939 14767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "b309f0babbfa4e21854dd9761d0ca6f7"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "e338a38e0b85b22449dcf4c58fdc4669"
server_key_iv: "b70d841b149f859d90565d59764dba52"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.371308 14767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "b309f0babbfa4e21854dd9761d0ca6f7"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "e338a38e0b85b22449dcf4c58fdc4669"
server_key_iv: "b70d841b149f859d90565d59764dba52"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.372605 14767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:58.373301 14782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.373477 14767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:56:58.373581 14767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "b309f0babbfa4e21854dd9761d0ca6f7"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "e338a38e0b85b22449dcf4c58fdc4669"
server_key_iv: "b70d841b149f859d90565d59764dba52"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.373653 14767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:58.381834 14767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:58.383615 14767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:58.383777 14767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:58.383988 14767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:58.384299 14767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:58.384353 14767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.384394 14767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:58.384423 14767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.390077 14767 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:46659
I20260929 18:56:58.390185 14895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:46659 every 8 connection(s)
I20260929 18:56:58.390491 14767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708218, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:56:58.399485 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.391195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:56937 (local address 127.29.71.190:39965)
0929 18:56:58.391274 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.391276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.391429 (+ 153us) server_negotiation.cc:407] Connection header received
0929 18:56:58.391897 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.391898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.391908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.391959 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:58.392813 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.393308 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.394384 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.394566 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.394620 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.396649 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:58.396650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:58.396651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:58.396658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:58.397914 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.398462 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.398464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.398464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.398522 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.398821 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.398822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.398822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.398942 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:58.399019 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.399305 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.399391 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:56:58.399668 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.391142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:39965 (local address 127.29.71.130:56937)
0929 18:56:58.391313 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.391348 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.391809 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.392011 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.392014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.392556 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:58.392739 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.392744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.393387 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.393388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.394304 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.394309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.394635 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.394636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.394733 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.395187 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:58.395195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:58.396542 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:58.398059 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:58.398066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:58.398068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:58.398370 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:58.398589 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:58.398590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:58.398590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:58.398712 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:58.399011 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:58.399017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:58.399158 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.399273 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.399430 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260929 18:56:58.400200 14896 heartbeater.cc:347] Connected to a master server at 127.29.71.190:39965
I20260929 18:56:58.400235 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 14767
I20260929 18:56:58.400313 14896 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:58.400352 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:56:58.400504 14896 heartbeater.cc:511] Master 127.29.71.190:39965 requested a full tablet report, sending...
I20260929 18:56:58.400537 29982 external_mini_cluster.cc:1428] Setting key c91289a421af980e63f6deefa5f66c43
I20260929 18:56:58.401044 14570 ts_manager.cc:194] Registered new tserver with Master: b309f0babbfa4e21854dd9761d0ca6f7 (127.29.71.130:46659)
I20260929 18:56:58.401464 14570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:56937
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:56:58.425244 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39965
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:56:58.506240 14901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:58.506485 14901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:58.506525 14901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:58.507697 14901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:58.507773 14901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:58.507819 14901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:56:58.509509 14901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:39965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:58.509804 14901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:58.510092 14901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:58.516007 14909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:58.516007 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:58.516037 14907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:58.516359 14901 server_base.cc:1034] running on GCE node
I20260929 18:56:58.516575 14901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:58.516806 14901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:58.517943 14901 hybrid_clock.cc:648] HybridClock initialized: now 1790708218517914 us; error 43 us; skew 500 ppm
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708218, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:56:58.520519 14901 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:56:58.521039 14901 webserver.cc:460] Webserver started at http://127.29.71.131:34299/ using document root <none> and password file <none>
I20260929 18:56:58.521286 14901 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:58.521353 14901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:58.521505 14901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:58.522635 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "84aab3cd409843b0957e573c63a54bba"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "f29fb2f76358d18d2655ec00931a890a"
server_key_iv: "c61a062b6c34e29e8318a23a0a866e18"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.523002 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "84aab3cd409843b0957e573c63a54bba"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "f29fb2f76358d18d2655ec00931a890a"
server_key_iv: "c61a062b6c34e29e8318a23a0a866e18"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.524354 14901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:58.525166 14916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.525518 14901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:58.525626 14901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "84aab3cd409843b0957e573c63a54bba"
format_stamp: "Formatted at 2026-09-29 18:56:58 on dist-test-slave-q2kh"
server_key: "f29fb2f76358d18d2655ec00931a890a"
server_key_iv: "c61a062b6c34e29e8318a23a0a866e18"
server_key_version: "encryptionkey@0"
I20260929 18:56:58.525705 14901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:58.536111 14901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:58.537977 14901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:58.538132 14901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:58.538439 14901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:58.538762 14901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:58.538815 14901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.538854 14901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:58.538882 14901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:58.544948 14901 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:42719
I20260929 18:56:58.545027 15029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:42719 every 8 connection(s)
I20260929 18:56:58.545346 14901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:56:58.550293 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 14901
I20260929 18:56:58.550411 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:56:58.550611 29982 external_mini_cluster.cc:1428] Setting key d8b598dd4972fba70c7fc62ab930a320
Sep 29 18:56:58 dist-test-slave-q2kh krb5kdc[14526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708218, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:56:58.555167 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.546094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:52981 (local address 127.29.71.190:39965)
0929 18:56:58.546198 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.546200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.546418 (+ 218us) server_negotiation.cc:407] Connection header received
0929 18:56:58.546946 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.546947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.546960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.547012 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:58.547964 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.548466 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.549467 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.549657 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.549738 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.552337 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:58.552338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:58.552338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:58.552346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:58.553662 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.554124 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.554125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.554126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.554247 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.554498 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.554499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.554499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.554619 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:58.554695 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.554994 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.555076 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:56:58.555380 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.546082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:39965 (local address 127.29.71.131:52981)
0929 18:56:58.546290 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.546326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.546830 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.547092 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.547096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.547655 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:58.547852 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.547857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.548560 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.548562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.549363 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.549366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.549748 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.549749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.549879 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.550524 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:56:58.550532 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:58.552235 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:58.553765 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:58.553772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:58.553774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:58.554015 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:58.554313 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:56:58.554313 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:56:58.554314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:56:58.554427 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:56:58.554686 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:58.554692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:58.554836 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.554967 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.555132 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
I20260929 18:56:58.555814 15030 heartbeater.cc:347] Connected to a master server at 127.29.71.190:39965
I20260929 18:56:58.555943 15030 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:58.556134 15030 heartbeater.cc:511] Master 127.29.71.190:39965 requested a full tablet report, sending...
I20260929 18:56:58.556560 14570 ts_manager.cc:194] Registered new tserver with Master: 84aab3cd409843b0957e573c63a54bba (127.29.71.131:42719)
I20260929 18:56:58.556953 14570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:52981
I20260929 18:56:58.562440 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:56:58.570099 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.563906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.29.71.190:39965)
0929 18:56:58.563975 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.563977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.564102 (+ 125us) server_negotiation.cc:407] Connection header received
0929 18:56:58.564141 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.564141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.564150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.564193 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:58.564786 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.565262 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.566030 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.566274 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.566341 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.567572 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:58.567573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:58.567573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:56:58.567579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:58.568996 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.569439 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.569440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.569440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.569491 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:56:58.569646 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:56:58.569647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:56:58.569647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:56:58.569777 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:58.569884 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.569974 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.570033 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:56:58.571977 14570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:56:58.573632 14570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:56:58.580657 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.576624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.29.71.131:42719)
0929 18:56:58.576908 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.576913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.576923 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:56:58.576967 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.576969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.576988 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.577283 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:58.577354 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.578135 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.579475 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.580228 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.580313 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.580384 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.580529 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.580574 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260929 18:56:58.580644 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.576455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:42719 (local address 127.0.0.1:38696)
0929 18:56:58.576715 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.576725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.576782 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.577101 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.577104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.577107 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:58.577185 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.577187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.578273 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.578274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.579378 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.579381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.580311 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.580312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.580392 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.580406 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.580500 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.580560 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"spinlock_wait_cycles":6016,"thread_start_us":81,"threads_started":1}
I20260929 18:56:58.580899 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.576455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46659 (local address 127.0.0.1:39052)
0929 18:56:58.576639 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.576684 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.576757 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.577308 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.577309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.577312 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:58.577387 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.577390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.578805 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.578806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.579767 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.579769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.580635 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.580637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.580702 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.580708 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.580738 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.580811 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":47,"thread_start_us":73,"threads_started":1}
I20260929 18:56:58.581183 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.576461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37501 (local address 127.0.0.1:59820)
0929 18:56:58.577089 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.577099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.577177 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.577425 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.577426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.577429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:58.577465 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.577467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.578935 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.578936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.579996 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.579999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.580958 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.580959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.581024 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.581028 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.581077 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.581119 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":77,"threads_started":1}
I20260929 18:56:58.581257 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.576731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.29.71.129:37501)
0929 18:56:58.577009 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.577014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.577150 (+ 136us) server_negotiation.cc:407] Connection header received
0929 18:56:58.577315 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.577318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.577332 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.577414 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:58.577527 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.578693 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.580086 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.580810 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.580872 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.580924 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.581117 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.581170 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
I20260929 18:56:58.581341 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.576918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.29.71.130:46659)
0929 18:56:58.577106 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.577109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.577119 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:56:58.577158 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.577160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.577174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.577464 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:58.577542 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.578703 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.579849 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.580532 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.581090 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.581163 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.581189 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.581240 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260929 18:56:58.581892 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da (DEFAULT_TABLE table=test-table [id=18155c15048e4a08a404e8e9dbfa729f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:58.582018 14830 tablet_service.cc:1467] Processing CreateTablet for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da (DEFAULT_TABLE table=test-table [id=18155c15048e4a08a404e8e9dbfa729f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:58.582288 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:58.582383 14830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:58.585067 15045 tablet_bootstrap.cc:492] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Bootstrap starting.
I20260929 18:56:58.585773 15045 tablet_bootstrap.cc:654] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:58.585661 14696 tablet_service.cc:1467] Processing CreateTablet for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da (DEFAULT_TABLE table=test-table [id=18155c15048e4a08a404e8e9dbfa729f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:58.586018 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:58.586323 15044 tablet_bootstrap.cc:492] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: Bootstrap starting.
I20260929 18:56:58.586400 15045 log.cc:824] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:58.587004 15044 tablet_bootstrap.cc:654] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:58.587245 15045 tablet_bootstrap.cc:492] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: No bootstrap required, opened a new log
I20260929 18:56:58.587324 15045 ts_tablet_manager.cc:1397] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:58.587555 15044 log.cc:824] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:58.588336 15044 tablet_bootstrap.cc:492] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: No bootstrap required, opened a new log
I20260929 18:56:58.588460 15044 ts_tablet_manager.cc:1397] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:58.589411 15048 tablet_bootstrap.cc:492] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d: Bootstrap starting.
I20260929 18:56:58.589373 15045 raft_consensus.cc:348] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.589565 15045 raft_consensus.cc:374] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:58.589630 15045 raft_consensus.cc:729] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84aab3cd409843b0957e573c63a54bba, State: Initialized, Role: FOLLOWER
I20260929 18:56:58.589771 15045 consensus_queue.cc:260] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.590068 15045 ts_tablet_manager.cc:1428] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:56:58.590090 15048 tablet_bootstrap.cc:654] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:58.590242 15030 heartbeater.cc:503] Master 127.29.71.190:39965 was elected leader, sending a full tablet report...
I20260929 18:56:58.590689 15048 log.cc:824] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:58.590618 15044 raft_consensus.cc:348] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.590776 15044 raft_consensus.cc:374] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:58.590857 15044 raft_consensus.cc:729] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b309f0babbfa4e21854dd9761d0ca6f7, State: Initialized, Role: FOLLOWER
I20260929 18:56:58.590991 15044 consensus_queue.cc:260] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.591261 15044 ts_tablet_manager.cc:1428] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:56:58.591401 14896 heartbeater.cc:503] Master 127.29.71.190:39965 was elected leader, sending a full tablet report...
I20260929 18:56:58.591499 15048 tablet_bootstrap.cc:492] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d: No bootstrap required, opened a new log
I20260929 18:56:58.591549 15048 ts_tablet_manager.cc:1397] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:58.593210 15048 raft_consensus.cc:348] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.593338 15048 raft_consensus.cc:374] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:58.593360 15048 raft_consensus.cc:729] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b6c8ebf2e040c3b7d17389a26fdc6d, State: Initialized, Role: FOLLOWER
I20260929 18:56:58.593465 15048 consensus_queue.cc:260] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.593689 15048 ts_tablet_manager.cc:1428] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:58.593770 14762 heartbeater.cc:503] Master 127.29.71.190:39965 was elected leader, sending a full tablet report...
W20260929 18:56:58.641084 14897 tablet.cc:2369] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:56:58.747430 14763 tablet.cc:2369] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:56:58.795920 15031 tablet.cc:2369] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:56:58.848901 15049 raft_consensus.cc:482] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:58.849076 15049 raft_consensus.cc:504] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.849503 15049 leader_election.cc:288] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b309f0babbfa4e21854dd9761d0ca6f7 (127.29.71.130:46659), e9b6c8ebf2e040c3b7d17389a26fdc6d (127.29.71.129:37501)
I20260929 18:56:58.853417 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.849740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37501 (local address 127.29.71.131:38233)
0929 18:56:58.849816 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.849827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.849873 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.850099 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.850100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.850102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:58.850224 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.850227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.851256 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.851256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.852535 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.852538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.853109 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.853110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.853202 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.853206 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.853260 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.853301 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":56}
I20260929 18:56:58.853412 15042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.849873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:38233 (local address 127.29.71.129:37501)
0929 18:56:58.849922 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.849924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.849934 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:56:58.849972 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.849973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.849980 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.850079 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:58.850369 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.851167 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.852642 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.853018 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.853079 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.853086 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.853291 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.853342 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 18:56:58.853488 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.849813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:47159 (local address 127.29.71.130:46659)
0929 18:56:58.849900 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.849902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.850055 (+ 153us) server_negotiation.cc:407] Connection header received
0929 18:56:58.850101 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.850102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.850112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.850195 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:58.850367 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.851125 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.852641 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.853097 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.853175 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.853185 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.853367 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.853412 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:56:58.853549 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.849806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46659 (local address 127.29.71.131:47159)
0929 18:56:58.849977 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.849986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:58.850030 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:58.850198 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:58.850200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:58.850201 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:58.850263 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.850265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.851245 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.851246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.852536 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:58.852540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.853190 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.853191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:58.853272 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.853277 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:56:58.853347 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 18:56:58.853387 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260929 18:56:58.853895 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e75f2a9e6bb4caa95a28c64d19ea8da" candidate_uuid: "84aab3cd409843b0957e573c63a54bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" is_pre_election: true
I20260929 18:56:58.853893 14850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e75f2a9e6bb4caa95a28c64d19ea8da" candidate_uuid: "84aab3cd409843b0957e573c63a54bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" is_pre_election: true
I20260929 18:56:58.854091 14716 raft_consensus.cc:2454] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84aab3cd409843b0957e573c63a54bba in term 0.
I20260929 18:56:58.854101 14850 raft_consensus.cc:2454] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84aab3cd409843b0957e573c63a54bba in term 0.
I20260929 18:56:58.854351 14917 leader_election.cc:302] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aab3cd409843b0957e573c63a54bba, b309f0babbfa4e21854dd9761d0ca6f7; no voters:
I20260929 18:56:58.854511 15049 raft_consensus.cc:2781] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:56:58.854571 15049 raft_consensus.cc:482] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:56:58.854595 15049 raft_consensus.cc:3037] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:58.855329 15049 raft_consensus.cc:504] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.855492 15049 leader_election.cc:288] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [CANDIDATE]: Term 1 election: Requested vote from peers b309f0babbfa4e21854dd9761d0ca6f7 (127.29.71.130:46659), e9b6c8ebf2e040c3b7d17389a26fdc6d (127.29.71.129:37501)
I20260929 18:56:58.855640 14850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e75f2a9e6bb4caa95a28c64d19ea8da" candidate_uuid: "84aab3cd409843b0957e573c63a54bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b309f0babbfa4e21854dd9761d0ca6f7"
I20260929 18:56:58.855707 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e75f2a9e6bb4caa95a28c64d19ea8da" candidate_uuid: "84aab3cd409843b0957e573c63a54bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d"
I20260929 18:56:58.855727 14850 raft_consensus.cc:3037] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:58.855809 14716 raft_consensus.cc:3037] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:58.856587 14850 raft_consensus.cc:2454] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84aab3cd409843b0957e573c63a54bba in term 1.
I20260929 18:56:58.856696 14716 raft_consensus.cc:2454] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84aab3cd409843b0957e573c63a54bba in term 1.
I20260929 18:56:58.856773 14917 leader_election.cc:302] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aab3cd409843b0957e573c63a54bba, b309f0babbfa4e21854dd9761d0ca6f7; no voters:
I20260929 18:56:58.856878 15049 raft_consensus.cc:2781] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:56:58.857038 15049 raft_consensus.cc:686] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 1 LEADER]: Becoming Leader. State: Replica: 84aab3cd409843b0957e573c63a54bba, State: Running, Role: LEADER
I20260929 18:56:58.857158 15049 consensus_queue.cc:237] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } }
I20260929 18:56:58.858325 14570 catalog_manager.cc:5495] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba reported cstate change: term changed from 0 to 1, leader changed from <none> to 84aab3cd409843b0957e573c63a54bba (127.29.71.131). New cstate: current_term: 1 leader_uuid: "84aab3cd409843b0957e573c63a54bba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84aab3cd409843b0957e573c63a54bba" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 42719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 } health_report { overall_health: UNKNOWN } } }
I20260929 18:56:58.933652 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.931124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.29.71.131:42719)
0929 18:56:58.931200 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:58.931202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:58.931213 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:56:58.931302 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:58.931303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:58.931329 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:58.931412 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:56:58.931566 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.932061 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.932851 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:58.933046 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:58.933123 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:58.933313 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:56:58.933418 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:56:58.933515 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:58.933534 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 18:56:58.933554 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:56:58.935842 14716 raft_consensus.cc:1261] T 1e75f2a9e6bb4caa95a28c64d19ea8da P e9b6c8ebf2e040c3b7d17389a26fdc6d [term 1 FOLLOWER]: Refusing update from remote peer 84aab3cd409843b0957e573c63a54bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:58.935858 14850 raft_consensus.cc:1261] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 1 FOLLOWER]: Refusing update from remote peer 84aab3cd409843b0957e573c63a54bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:58.936264 15049 consensus_queue.cc:1035] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b6c8ebf2e040c3b7d17389a26fdc6d" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:58.936465 15054 consensus_queue.cc:1035] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b309f0babbfa4e21854dd9761d0ca6f7" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:58.938731 15057 mvcc.cc:205] Tried to move back new op lower bound from 7334740864757489664 to 7334740864439676928. Current Snapshot: MvccSnapshot[applied={T|T < 7334740864757489664}]
I20260929 18:56:58.938933 15059 mvcc.cc:205] Tried to move back new op lower bound from 7334740864757489664 to 7334740864439676928. Current Snapshot: MvccSnapshot[applied={T|T < 7334740864757489664}]
I20260929 18:56:58.939441 15058 mvcc.cc:205] Tried to move back new op lower bound from 7334740864757489664 to 7334740864439676928. Current Snapshot: MvccSnapshot[applied={T|T < 7334740864757489664}]
I20260929 18:56:58.941728 14570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:58.941833 14570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:58.942626 14570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07ea429d67f840dc9ed5087b058636f3: Sending DeleteTablet for 3 replicas of tablet 1e75f2a9e6bb4caa95a28c64d19ea8da
I20260929 18:56:58.942957 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:58 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:38696
I20260929 18:56:58.942999 14696 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:58 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:59820
I20260929 18:56:58.943001 14830 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:58 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:39052
I20260929 18:56:58.943235 15066 tablet_replica.cc:330] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: stopping tablet replica
I20260929 18:56:58.943337 15066 raft_consensus.cc:2229] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:56:58.943514 15066 raft_consensus.cc:2258] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:58.943552 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 14633
I20260929 18:56:58.944499 15066 ts_tablet_manager.cc:1905] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:56:58.946732 15066 ts_tablet_manager.cc:1918] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:58.946801 15066 log.cc:1196] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/1e75f2a9e6bb4caa95a28c64d19ea8da
I20260929 18:56:58.947115 15066 ts_tablet_manager.cc:1939] T 1e75f2a9e6bb4caa95a28c64d19ea8da P 84aab3cd409843b0957e573c63a54bba: Deleting consensus metadata
I20260929 18:56:58.947580 14556 catalog_manager.cc:4841] TS 84aab3cd409843b0957e573c63a54bba (127.29.71.131:42719): tablet 1e75f2a9e6bb4caa95a28c64d19ea8da (table test-table [id=18155c15048e4a08a404e8e9dbfa729f]) successfully deleted
W20260929 18:56:58.950205 14558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:37501 (error 108)
I20260929 18:56:58.950379 15067 tablet_replica.cc:330] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7: stopping tablet replica
I20260929 18:56:58.950503 15067 raft_consensus.cc:2229] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:56:58.950627 15067 raft_consensus.cc:2258] T 1e75f2a9e6bb4caa95a28c64d19ea8da P b309f0babbfa4e21854dd9761d0ca6f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:58.950709 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.950591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37501 (local address 127.0.0.1:59832)
0929 18:56:58.950639 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.950677 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:56:58.950824 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 14767
W20260929 18:56:58.950860 14558 catalog_manager.cc:4584] TS e9b6c8ebf2e040c3b7d17389a26fdc6d (127.29.71.129:37501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e75f2a9e6bb4caa95a28c64d19ea8da: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37501: connect: Connection refused (error 111)
I20260929 18:56:58.956120 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:58.955992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46659 (local address 127.0.0.1:39058)
0929 18:56:58.956051 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:58.956080 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.130:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 18:56:58.956218 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 14901
I20260929 18:56:58.960583 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 14539
2026-09-29T18:56:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2261 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T18:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:56:58Z Disabled control of system clock
I20260929 18:56:58.988191 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:44939
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:56:59.068936 15073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:59.069142 15073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:59.069161 15073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:59.070241 15073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:56:59.070271 15073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:59.070284 15073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:59.070297 15073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:56:59.070308 15073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:56:59.071583 15073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:44939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:44939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:59.071763 15073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:59.071979 15073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:59.076913 15081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:59.076906 15078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:59.076906 15079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:59.077018 15073 server_base.cc:1034] running on GCE node
I20260929 18:56:59.077438 15073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:59.077725 15073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:59.078876 15073 hybrid_clock.cc:648] HybridClock initialized: now 1790708219078858 us; error 30 us; skew 500 ppm
I20260929 18:56:59.080229 15073 webserver.cc:460] Webserver started at http://127.29.71.190:39511/ using document root <none> and password file <none>
I20260929 18:56:59.080497 15073 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:59.080566 15073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:59.080713 15073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:59.081899 15073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "eea1f14238694f2e93b9ece151ce5570"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "f23aff2c42fef360a8bf6b7b2983d37e"
server_key_iv: "4f162b3dc5ffaca3e25fffbf69659731"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.082317 15073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "eea1f14238694f2e93b9ece151ce5570"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "f23aff2c42fef360a8bf6b7b2983d37e"
server_key_iv: "4f162b3dc5ffaca3e25fffbf69659731"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.083609 15073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:56:59.084348 15087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.084524 15073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:56:59.084641 15073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "eea1f14238694f2e93b9ece151ce5570"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "f23aff2c42fef360a8bf6b7b2983d37e"
server_key_iv: "4f162b3dc5ffaca3e25fffbf69659731"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.084729 15073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:59.099489 15073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:59.099829 15073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:59.099989 15073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:59.103718 15073 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:44939
I20260929 18:56:59.103775 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:44939 every 8 connection(s)
I20260929 18:56:59.104167 15073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:56:59.105199 15140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:59.107510 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570: Bootstrap starting.
I20260929 18:56:59.108089 15140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:59.108384 15140 log.cc:824] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:59.109074 15140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570: No bootstrap required, opened a new log
I20260929 18:56:59.110679 15140 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea1f14238694f2e93b9ece151ce5570" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44939 } }
I20260929 18:56:59.110822 15140 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:59.110845 15140 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea1f14238694f2e93b9ece151ce5570, State: Initialized, Role: FOLLOWER
I20260929 18:56:59.110994 15140 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea1f14238694f2e93b9ece151ce5570" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44939 } }
I20260929 18:56:59.111068 15140 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:56:59.111089 15140 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:56:59.111116 15140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:59.111598 15140 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea1f14238694f2e93b9ece151ce5570" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44939 } }
I20260929 18:56:59.111703 15140 leader_election.cc:302] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea1f14238694f2e93b9ece151ce5570; no voters:
I20260929 18:56:59.111894 15140 leader_election.cc:288] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:56:59.111945 15143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:56:59.112161 15140 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:56:59.112165 15143 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [term 1 LEADER]: Becoming Leader. State: Replica: eea1f14238694f2e93b9ece151ce5570, State: Running, Role: LEADER
I20260929 18:56:59.112437 15143 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea1f14238694f2e93b9ece151ce5570" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44939 } }
I20260929 18:56:59.112855 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15073
I20260929 18:56:59.112988 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:56:59.112973 15144 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea1f14238694f2e93b9ece151ce5570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea1f14238694f2e93b9ece151ce5570" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44939 } } }
I20260929 18:56:59.113054 15144 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:59.113232 29982 external_mini_cluster.cc:1428] Setting key d810d50668d4d94a8295415103a9f954
I20260929 18:56:59.113343 15152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:56:59.113408 15145 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea1f14238694f2e93b9ece151ce5570. Latest consensus state: current_term: 1 leader_uuid: "eea1f14238694f2e93b9ece151ce5570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea1f14238694f2e93b9ece151ce5570" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 44939 } } }
I20260929 18:56:59.113484 15145 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570 [sys.catalog]: This master's current role is: LEADER
I20260929 18:56:59.114130 15152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:56:59.115782 15152 catalog_manager.cc:1347] Generated new cluster ID: 60dfe1f8d56b41879977b19f535a11d9
I20260929 18:56:59.115834 15152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:56:59.119395 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.113824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.29.71.190:44939)
0929 18:56:59.114033 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.114042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.114073 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:56:59.114204 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.114211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.114278 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.114644 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:59.114749 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.116115 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.117238 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.117865 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.117993 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.118690 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:59.118694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:59.118694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:56:59.118904 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:59.118946 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:56:59.118960 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:59.119031 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.119099 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.119178 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260929 18:56:59.137243 15152 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:56:59.137799 15152 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:56:59.144032 15152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570: Generated new TSK 0
I20260929 18:56:59.144264 15152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:56:59.150147 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44939
--builtin_ntp_servers=127.29.71.148:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:56:59.234380 15164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:59.234578 15164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:59.234601 15164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:59.235816 15164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:59.235857 15164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:59.235894 15164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:56:59.237291 15164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:59.237565 15164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:59.237890 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:59.243805 15170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:59.243814 15169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:59.243952 15164 server_base.cc:1034] running on GCE node
W20260929 18:56:59.243894 15172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:59.244244 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:59.244496 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:59.245646 15164 hybrid_clock.cc:648] HybridClock initialized: now 1790708219245604 us; error 52 us; skew 500 ppm
I20260929 18:56:59.247100 15164 webserver.cc:460] Webserver started at http://127.29.71.129:43439/ using document root <none> and password file <none>
I20260929 18:56:59.247352 15164 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:59.247402 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:59.247534 15164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:59.248639 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "21c495556d0c4363ba700543fa8727ec"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "9d842b2b48f7e9d34b29d0cea30eb168"
server_key_iv: "91712a754cc75526b34384ceba984fe9"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.249022 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "21c495556d0c4363ba700543fa8727ec"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "9d842b2b48f7e9d34b29d0cea30eb168"
server_key_iv: "91712a754cc75526b34384ceba984fe9"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.250639 15164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:56:59.251534 15178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.251739 15164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:59.251875 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "21c495556d0c4363ba700543fa8727ec"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "9d842b2b48f7e9d34b29d0cea30eb168"
server_key_iv: "91712a754cc75526b34384ceba984fe9"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.251960 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:59.259634 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:59.259950 15164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:59.260111 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:59.260349 15164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:59.260675 15164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:59.260732 15164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.260773 15164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:59.260803 15164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.266520 15164 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:39729
I20260929 18:56:59.266608 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:39729 every 8 connection(s)
I20260929 18:56:59.266942 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:56:59.271974 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.267693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:51867 (local address 127.29.71.190:44939)
0929 18:56:59.267758 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.267760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.268009 (+ 249us) server_negotiation.cc:407] Connection header received
0929 18:56:59.268477 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.268477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.268488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.268546 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:59.269191 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.269681 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.270739 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.270924 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.270979 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.271753 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:59.271754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:59.271754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:56:59.271759 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:59.271773 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:56:59.271777 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:59.271818 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.271883 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.271911 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:56:59.272181 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.267655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44939 (local address 127.29.71.129:51867)
0929 18:56:59.267880 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.267921 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.268396 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.268606 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.268609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.268617 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:59.269121 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.269125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.269799 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.269799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.270659 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.270661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.271000 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.271001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.271116 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.271612 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:56:59.271613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:59.271644 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:56:59.271668 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:59.271819 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:59.271821 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:59.271824 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.271873 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.271952 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260929 18:56:59.272691 15292 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44939
I20260929 18:56:59.272830 15292 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:59.272981 15292 heartbeater.cc:511] Master 127.29.71.190:44939 requested a full tablet report, sending...
I20260929 18:56:59.273506 15104 ts_manager.cc:194] Registered new tserver with Master: 21c495556d0c4363ba700543fa8727ec (127.29.71.129:39729)
I20260929 18:56:59.274073 15104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.129:51867
I20260929 18:56:59.276479 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15164
I20260929 18:56:59.276851 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:56:59.277066 29982 external_mini_cluster.cc:1428] Setting key b7ae010162ddc3f96103fae489249b42
I20260929 18:56:59.277971 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44939
--builtin_ntp_servers=127.29.71.148:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:56:59.358383 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:59.358582 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:59.358604 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:59.359673 15295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:59.359706 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:59.359732 15295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:56:59.360982 15295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:59.361161 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:59.361399 15295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:59.366357 15301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:59.366400 15303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:59.366376 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:59.366644 15295 server_base.cc:1034] running on GCE node
I20260929 18:56:59.366945 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:59.367208 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:59.368350 15295 hybrid_clock.cc:648] HybridClock initialized: now 1790708219368319 us; error 45 us; skew 500 ppm
I20260929 18:56:59.369586 15295 webserver.cc:460] Webserver started at http://127.29.71.130:35303/ using document root <none> and password file <none>
I20260929 18:56:59.369827 15295 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:59.369908 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:59.370055 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:59.371192 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "18a4a04f465143d3941d97740ed8eedb"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "faeec99233041a6266d7046ad8a95916"
server_key_iv: "d79a1c62e05eb15bfbb5750aaeb430d5"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.371556 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "18a4a04f465143d3941d97740ed8eedb"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "faeec99233041a6266d7046ad8a95916"
server_key_iv: "d79a1c62e05eb15bfbb5750aaeb430d5"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.372936 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:56:59.373669 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.373881 15295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:56:59.373978 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "18a4a04f465143d3941d97740ed8eedb"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "faeec99233041a6266d7046ad8a95916"
server_key_iv: "d79a1c62e05eb15bfbb5750aaeb430d5"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.374055 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:59.390610 15295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:59.390930 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:59.391083 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:59.391324 15295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:59.391666 15295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:59.391717 15295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.391752 15295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:59.391778 15295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.397281 15295 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:37107
I20260929 18:56:59.397363 15422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:37107 every 8 connection(s)
I20260929 18:56:59.397702 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:56:59.402732 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15295
I20260929 18:56:59.402851 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:56:59.403097 29982 external_mini_cluster.cc:1428] Setting key d0c4e3b8192e30484cfd2e40f283733c
I20260929 18:56:59.403200 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.398514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:47415 (local address 127.29.71.190:44939)
0929 18:56:59.398559 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.398561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.398782 (+ 221us) server_negotiation.cc:407] Connection header received
0929 18:56:59.399341 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.399342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.399354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.399403 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:59.400083 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.400581 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.401588 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.401789 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.401875 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.402878 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:59.402879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:59.402880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:56:59.402885 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:59.402905 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:56:59.402910 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:59.402975 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.403071 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.403096 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:56:59.403486 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.398482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44939 (local address 127.29.71.130:47415)
0929 18:56:59.398660 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.398695 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.399228 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.399475 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.399479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.399489 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:59.399983 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.399988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.400677 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.400678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.401479 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.401481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.401875 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.401876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.402046 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.402677 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:56:59.402677 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:59.402724 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:56:59.402751 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:59.402980 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:59.402987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:59.402990 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.403070 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.403188 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260929 18:56:59.404017 15423 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44939
I20260929 18:56:59.404134 15423 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:59.404134 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44939
--builtin_ntp_servers=127.29.71.148:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:56:59.404352 15423 heartbeater.cc:511] Master 127.29.71.190:44939 requested a full tablet report, sending...
I20260929 18:56:59.404748 15104 ts_manager.cc:194] Registered new tserver with Master: 18a4a04f465143d3941d97740ed8eedb (127.29.71.130:37107)
I20260929 18:56:59.405125 15104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.130:47415
W20260929 18:56:59.485455 15426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:56:59.485663 15426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:56:59.485684 15426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:56:59.486997 15426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:56:59.487038 15426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:56:59.487064 15426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:56:59.488511 15426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:44939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:56:59.488763 15426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:56:59.489068 15426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:56:59.493995 15434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:56:59.493995 15431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:59.494212 15426 server_base.cc:1034] running on GCE node
W20260929 18:56:59.494050 15432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:56:59.494518 15426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:56:59.494783 15426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:56:59.495949 15426 hybrid_clock.cc:648] HybridClock initialized: now 1790708219495941 us; error 41 us; skew 500 ppm
I20260929 18:56:59.497247 15426 webserver.cc:460] Webserver started at http://127.29.71.131:34293/ using document root <none> and password file <none>
I20260929 18:56:59.497514 15426 fs_manager.cc:359] Metadata directory not provided
I20260929 18:56:59.497586 15426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:56:59.497740 15426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:56:59.498926 15426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "afcb8723959548a89e009d6c72dc4b9a"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "b433f8339fecc29800f5e492f3babb79"
server_key_iv: "6efdc2fb782f3cb61792d7de20e614fd"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.499286 15426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "afcb8723959548a89e009d6c72dc4b9a"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "b433f8339fecc29800f5e492f3babb79"
server_key_iv: "6efdc2fb782f3cb61792d7de20e614fd"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.500583 15426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:56:59.501343 15440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.501643 15426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:56:59.501745 15426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "afcb8723959548a89e009d6c72dc4b9a"
format_stamp: "Formatted at 2026-09-29 18:56:59 on dist-test-slave-q2kh"
server_key: "b433f8339fecc29800f5e492f3babb79"
server_key_iv: "6efdc2fb782f3cb61792d7de20e614fd"
server_key_version: "encryptionkey@0"
I20260929 18:56:59.501818 15426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:56:59.522365 15426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:56:59.522677 15426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:56:59.522826 15426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:56:59.523065 15426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:56:59.523408 15426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:56:59.523463 15426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.523504 15426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:56:59.523530 15426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:56:59.529320 15426 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:45395
I20260929 18:56:59.529383 15553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:45395 every 8 connection(s)
I20260929 18:56:59.529749 15426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:56:59.530117 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15426
I20260929 18:56:59.530270 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:56:59.530498 29982 external_mini_cluster.cc:1428] Setting key 9e19d219b5c6e8b22adfceb8d9909153
I20260929 18:56:59.535388 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.530573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:36383 (local address 127.29.71.190:44939)
0929 18:56:59.530627 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.530630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.530974 (+ 344us) server_negotiation.cc:407] Connection header received
0929 18:56:59.531699 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.531700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.531711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.531757 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:59.532441 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.532940 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.533919 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.534130 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.534242 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.535079 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:59.535080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:59.535081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:56:59.535086 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:59.535101 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:56:59.535105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:59.535177 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.535271 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.535300 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 18:56:59.535616 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.530581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:44939 (local address 127.29.71.131:36383)
0929 18:56:59.530855 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.530908 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.531601 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.531823 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.531827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.531836 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:56:59.532339 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.532343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.533029 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.533030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.533803 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.533805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.534252 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.534253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.534388 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.534930 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:56:59.534930 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:56:59.534957 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:56:59.534979 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:56:59.535180 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:56:59.535185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:56:59.535189 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.535275 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.535371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":154,"threads_started":1}
I20260929 18:56:59.536090 15554 heartbeater.cc:347] Connected to a master server at 127.29.71.190:44939
I20260929 18:56:59.536162 15554 heartbeater.cc:464] Registering TS with master...
I20260929 18:56:59.536355 15554 heartbeater.cc:511] Master 127.29.71.190:44939 requested a full tablet report, sending...
I20260929 18:56:59.536751 15104 ts_manager.cc:194] Registered new tserver with Master: afcb8723959548a89e009d6c72dc4b9a (127.29.71.131:45395)
I20260929 18:56:59.537113 15104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.131:36383
I20260929 18:56:59.542117 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:56:59.546334 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.543583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.29.71.190:44939)
0929 18:56:59.543646 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.543647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.543657 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:56:59.543722 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.543722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.543734 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.543791 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:56:59.543959 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.544420 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.545174 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.545380 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.545462 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.546085 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:56:59.546087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:56:59.546087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:56:59.546092 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:56:59.546105 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:56:59.546109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:56:59.546211 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.546227 (+ 16us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.546249 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:56:59.547785 15104 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:56:59.549448 15104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:56:59.555970 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.551623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.29.71.130:37107)
0929 18:56:59.551838 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.551843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.551853 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:56:59.551909 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.551912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.551930 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.552009 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:59.552234 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.553363 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.554844 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.555556 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.555623 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.555709 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.555847 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.555887 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260929 18:56:59.555972 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.551511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:37107 (local address 127.0.0.1:55586)
0929 18:56:59.551625 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.551654 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.551728 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.552082 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.552084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.552086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:59.552160 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.552162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.553451 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.553452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.554773 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.554775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.555647 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.555648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.555731 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.555744 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.555818 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.555882 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":27,"thread_start_us":78,"threads_started":1}
I20260929 18:56:59.555972 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.551898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:39729 (local address 127.0.0.1:48944)
0929 18:56:59.552271 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.552279 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.552328 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.552658 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.552659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.552660 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:59.552690 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.552691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.553562 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.553563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.554666 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.554668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.555649 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.555649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.555731 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.555744 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.555818 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.555894 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":6,"thread_start_us":61,"threads_started":1}
I20260929 18:56:59.556195 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.552022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.29.71.129:39729)
0929 18:56:59.552204 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.552209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.552454 (+ 245us) server_negotiation.cc:407] Connection header received
0929 18:56:59.552500 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.552502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.552518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.552588 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:59.552762 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.553476 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.554751 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.555550 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.555622 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.555709 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.556078 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.556117 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260929 18:56:59.556954 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.551725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:45395 (local address 127.0.0.1:55636)
0929 18:56:59.552103 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.552113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.552168 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.552422 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.552423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.552425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:59.552468 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.552469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.553671 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.553672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.554772 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.554775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.556712 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.556713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.556759 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.556762 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.556837 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.556886 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":74,"threads_started":1}
I20260929 18:56:59.556967 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.551833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.29.71.131:45395)
0929 18:56:59.552044 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.552049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.552223 (+ 174us) server_negotiation.cc:407] Connection header received
0929 18:56:59.552264 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.552267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.552282 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.552349 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:59.552544 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.553599 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.555745 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.556458 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.556557 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.556645 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.556854 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.556893 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260929 18:56:59.557144 15357 tablet_service.cc:1467] Processing CreateTablet for tablet befadae4a5a74c76a76614d47e775179 (DEFAULT_TABLE table=test-table [id=c4d51497f1b04114ada7621228118c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:59.557500 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befadae4a5a74c76a76614d47e775179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:59.557843 15488 tablet_service.cc:1467] Processing CreateTablet for tablet befadae4a5a74c76a76614d47e775179 (DEFAULT_TABLE table=test-table [id=c4d51497f1b04114ada7621228118c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:59.558135 15488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befadae4a5a74c76a76614d47e775179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:59.560210 15568 tablet_bootstrap.cc:492] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Bootstrap starting.
I20260929 18:56:59.560607 15226 tablet_service.cc:1467] Processing CreateTablet for tablet befadae4a5a74c76a76614d47e775179 (DEFAULT_TABLE table=test-table [id=c4d51497f1b04114ada7621228118c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:56:59.560727 15569 tablet_bootstrap.cc:492] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Bootstrap starting.
I20260929 18:56:59.560917 15568 tablet_bootstrap.cc:654] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:59.560942 15226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befadae4a5a74c76a76614d47e775179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:56:59.561218 15568 log.cc:824] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:59.561257 15569 tablet_bootstrap.cc:654] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:59.561515 15569 log.cc:824] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:59.561997 15568 tablet_bootstrap.cc:492] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: No bootstrap required, opened a new log
I20260929 18:56:59.562057 15568 ts_tablet_manager.cc:1397] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:56:59.562111 15569 tablet_bootstrap.cc:492] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: No bootstrap required, opened a new log
I20260929 18:56:59.562203 15569 ts_tablet_manager.cc:1397] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:56:59.564265 15572 tablet_bootstrap.cc:492] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec: Bootstrap starting.
I20260929 18:56:59.564416 15569 raft_consensus.cc:348] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.564584 15569 raft_consensus.cc:374] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:59.564627 15569 raft_consensus.cc:729] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcb8723959548a89e009d6c72dc4b9a, State: Initialized, Role: FOLLOWER
I20260929 18:56:59.564779 15569 consensus_queue.cc:260] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.564805 15568 raft_consensus.cc:348] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.564958 15572 tablet_bootstrap.cc:654] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec: Neither blocks nor log segments found. Creating new log.
I20260929 18:56:59.564992 15568 raft_consensus.cc:374] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:59.565037 15568 raft_consensus.cc:729] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a4a04f465143d3941d97740ed8eedb, State: Initialized, Role: FOLLOWER
I20260929 18:56:59.565045 15569 ts_tablet_manager.cc:1428] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:56:59.565174 15554 heartbeater.cc:503] Master 127.29.71.190:44939 was elected leader, sending a full tablet report...
I20260929 18:56:59.565160 15568 consensus_queue.cc:260] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.565246 15572 log.cc:824] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec: Log is configured to *not* fsync() on all Append() calls
I20260929 18:56:59.565418 15568 ts_tablet_manager.cc:1428] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:56:59.565471 15423 heartbeater.cc:503] Master 127.29.71.190:44939 was elected leader, sending a full tablet report...
I20260929 18:56:59.566093 15572 tablet_bootstrap.cc:492] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec: No bootstrap required, opened a new log
I20260929 18:56:59.566196 15572 ts_tablet_manager.cc:1397] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:59.568133 15572 raft_consensus.cc:348] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.568253 15572 raft_consensus.cc:374] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:56:59.568274 15572 raft_consensus.cc:729] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c495556d0c4363ba700543fa8727ec, State: Initialized, Role: FOLLOWER
I20260929 18:56:59.568394 15572 consensus_queue.cc:260] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.568631 15572 ts_tablet_manager.cc:1428] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:56:59.568686 15292 heartbeater.cc:503] Master 127.29.71.190:44939 was elected leader, sending a full tablet report...
I20260929 18:56:59.642414 15573 raft_consensus.cc:482] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:56:59.642532 15573 raft_consensus.cc:504] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.642879 15573 leader_election.cc:288] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a4a04f465143d3941d97740ed8eedb (127.29.71.130:37107), 21c495556d0c4363ba700543fa8727ec (127.29.71.129:39729)
I20260929 18:56:59.646279 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.643424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:59159 (local address 127.29.71.129:39729)
0929 18:56:59.643485 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.643487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.643570 (+ 83us) server_negotiation.cc:407] Connection header received
0929 18:56:59.643673 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.643674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.643686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.643742 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:59.643881 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.644582 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.645527 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.645890 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.645944 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.645952 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.646135 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.646206 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:56:59.646268 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.643277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:39729 (local address 127.29.71.131:59159)
0929 18:56:59.643513 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.643523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.643594 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.643771 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.643772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.643773 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:59.643819 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.643820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.644654 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.644656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.645419 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.645422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.645976 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.645977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.646049 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.646053 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.646106 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.646137 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260929 18:56:59.646328 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.643249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:42817 (local address 127.29.71.130:37107)
0929 18:56:59.643297 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.643313 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.643321 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:56:59.643351 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.643352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.643361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.643436 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:56:59.643699 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.644476 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.645509 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.645898 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.645994 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.646001 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.646206 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.646250 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:56:59.646392 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.643160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:37107 (local address 127.29.71.131:42817)
0929 18:56:59.643202 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.643216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:56:59.643259 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:56:59.643535 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:56:59.643537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:56:59.643539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:56:59.643617 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.643619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.644574 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.644575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.645409 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:56:59.645411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.646012 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.646013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:56:59.646085 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.646107 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:56:59.646192 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:56:59.646233 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 18:56:59.646673 15246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befadae4a5a74c76a76614d47e775179" candidate_uuid: "afcb8723959548a89e009d6c72dc4b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c495556d0c4363ba700543fa8727ec" is_pre_election: true
I20260929 18:56:59.646684 15377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befadae4a5a74c76a76614d47e775179" candidate_uuid: "afcb8723959548a89e009d6c72dc4b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a4a04f465143d3941d97740ed8eedb" is_pre_election: true
I20260929 18:56:59.646881 15246 raft_consensus.cc:2454] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afcb8723959548a89e009d6c72dc4b9a in term 0.
I20260929 18:56:59.646891 15377 raft_consensus.cc:2454] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afcb8723959548a89e009d6c72dc4b9a in term 0.
I20260929 18:56:59.647109 15441 leader_election.cc:302] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a4a04f465143d3941d97740ed8eedb, afcb8723959548a89e009d6c72dc4b9a; no voters:
I20260929 18:56:59.647270 15573 raft_consensus.cc:2781] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:56:59.647349 15573 raft_consensus.cc:482] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:56:59.647370 15573 raft_consensus.cc:3037] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:59.648118 15573 raft_consensus.cc:504] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.648262 15573 leader_election.cc:288] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [CANDIDATE]: Term 1 election: Requested vote from peers 18a4a04f465143d3941d97740ed8eedb (127.29.71.130:37107), 21c495556d0c4363ba700543fa8727ec (127.29.71.129:39729)
W20260929 18:56:59.648247 15424 tablet.cc:2369] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:56:59.648422 15246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befadae4a5a74c76a76614d47e775179" candidate_uuid: "afcb8723959548a89e009d6c72dc4b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c495556d0c4363ba700543fa8727ec"
I20260929 18:56:59.648471 15377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befadae4a5a74c76a76614d47e775179" candidate_uuid: "afcb8723959548a89e009d6c72dc4b9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a4a04f465143d3941d97740ed8eedb"
I20260929 18:56:59.648504 15246 raft_consensus.cc:3037] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:59.648553 15377 raft_consensus.cc:3037] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:56:59.649267 15246 raft_consensus.cc:2454] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afcb8723959548a89e009d6c72dc4b9a in term 1.
I20260929 18:56:59.649294 15377 raft_consensus.cc:2454] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afcb8723959548a89e009d6c72dc4b9a in term 1.
I20260929 18:56:59.649446 15441 leader_election.cc:302] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a4a04f465143d3941d97740ed8eedb, afcb8723959548a89e009d6c72dc4b9a; no voters:
I20260929 18:56:59.649585 15573 raft_consensus.cc:2781] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:56:59.649775 15573 raft_consensus.cc:686] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 1 LEADER]: Becoming Leader. State: Replica: afcb8723959548a89e009d6c72dc4b9a, State: Running, Role: LEADER
I20260929 18:56:59.649884 15573 consensus_queue.cc:237] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } }
I20260929 18:56:59.650820 15104 catalog_manager.cc:5495] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a reported cstate change: term changed from 0 to 1, leader changed from <none> to afcb8723959548a89e009d6c72dc4b9a (127.29.71.131). New cstate: current_term: 1 leader_uuid: "afcb8723959548a89e009d6c72dc4b9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afcb8723959548a89e009d6c72dc4b9a" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 } health_report { overall_health: UNKNOWN } } }
I20260929 18:56:59.671460 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.668961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.29.71.131:45395)
0929 18:56:59.669030 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 18:56:59.669031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:56:59.669038 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:56:59.669074 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:56:59.669074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:56:59.669119 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:56:59.669190 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:56:59.669327 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.669794 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.670652 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:56:59.670849 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:56:59.670936 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:56:59.671098 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:56:59.671207 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:56:59.671306 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:56:59.671325 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 18:56:59.671345 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:56:59.673439 15246 raft_consensus.cc:1261] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 1 FOLLOWER]: Refusing update from remote peer afcb8723959548a89e009d6c72dc4b9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:59.673439 15377 raft_consensus.cc:1261] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 1 FOLLOWER]: Refusing update from remote peer afcb8723959548a89e009d6c72dc4b9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:56:59.673883 15573 consensus_queue.cc:1035] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a4a04f465143d3941d97740ed8eedb" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:59.674080 15578 consensus_queue.cc:1035] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c495556d0c4363ba700543fa8727ec" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:56:59.677627 15583 mvcc.cc:205] Tried to move back new op lower bound from 7334740867779518464 to 7334740867686756352. Current Snapshot: MvccSnapshot[applied={T|T < 7334740867779518464 or (T in {7334740867779518464})}]
I20260929 18:56:59.679361 15097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:59.679464 15097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:56:59.680233 15097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eea1f14238694f2e93b9ece151ce5570: Sending DeleteTablet for 3 replicas of tablet befadae4a5a74c76a76614d47e775179
I20260929 18:56:59.680598 15357 tablet_service.cc:1514] Processing DeleteTablet for tablet befadae4a5a74c76a76614d47e775179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:59 UTC) from {username='slave'} at 127.0.0.1:55586
I20260929 18:56:59.680703 15226 tablet_service.cc:1514] Processing DeleteTablet for tablet befadae4a5a74c76a76614d47e775179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:59 UTC) from {username='slave'} at 127.0.0.1:48944
I20260929 18:56:59.680812 15488 tablet_service.cc:1514] Processing DeleteTablet for tablet befadae4a5a74c76a76614d47e775179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:56:59 UTC) from {username='slave'} at 127.0.0.1:55636
I20260929 18:56:59.680857 15592 tablet_replica.cc:330] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: stopping tablet replica
I20260929 18:56:59.680961 15592 raft_consensus.cc:2229] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:56:59.681070 15592 raft_consensus.cc:2258] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:59.681071 15594 tablet_replica.cc:330] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: stopping tablet replica
I20260929 18:56:59.681159 15594 raft_consensus.cc:2229] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:56:59.681319 15594 raft_consensus.cc:2258] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:59.681321 15593 tablet_replica.cc:330] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec: stopping tablet replica
I20260929 18:56:59.681396 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15164
I20260929 18:56:59.681420 15593 raft_consensus.cc:2229] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:56:59.681515 15593 raft_consensus.cc:2258] T befadae4a5a74c76a76614d47e775179 P 21c495556d0c4363ba700543fa8727ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:56:59.682452 15594 ts_tablet_manager.cc:1905] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:56:59.684626 15594 ts_tablet_manager.cc:1918] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:59.684746 15594 log.cc:1196] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/befadae4a5a74c76a76614d47e775179
I20260929 18:56:59.685048 15594 ts_tablet_manager.cc:1939] T befadae4a5a74c76a76614d47e775179 P afcb8723959548a89e009d6c72dc4b9a: Deleting consensus metadata
I20260929 18:56:59.685495 15088 catalog_manager.cc:4841] TS afcb8723959548a89e009d6c72dc4b9a (127.29.71.131:45395): tablet befadae4a5a74c76a76614d47e775179 (table test-table [id=c4d51497f1b04114ada7621228118c2c]) successfully deleted
I20260929 18:56:59.687041 15592 ts_tablet_manager.cc:1905] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:56:59.688302 15088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:39729 (error 108)
I20260929 18:56:59.688899 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15295
I20260929 18:56:59.689193 15592 ts_tablet_manager.cc:1918] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:56:59.689296 15592 log.cc:1196] T befadae4a5a74c76a76614d47e775179 P 18a4a04f465143d3941d97740ed8eedb: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/befadae4a5a74c76a76614d47e775179
I20260929 18:56:59.694933 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15426
I20260929 18:56:59.695668 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.695099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:39729 (local address 127.0.0.1:48960)
0929 18:56:59.695591 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.695635 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:39729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470}
I20260929 18:56:59.695793 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:56:59.695169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:37107 (local address 127.0.0.1:55588)
0929 18:56:59.695757 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 18:56:59.695779 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.130:37107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
W20260929 18:56:59.695842 15088 catalog_manager.cc:4584] TS 21c495556d0c4363ba700543fa8727ec (127.29.71.129:39729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet befadae4a5a74c76a76614d47e775179: Network error: Client connection negotiation failed: client connection to 127.29.71.129:39729: connect: Connection refused (error 111)
I20260929 18:56:59.700028 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15073
2026-09-29T18:56:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (739 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:56:59 dist-test-slave-q2kh krb5kdc[15598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:56:59 dist-test-slave-q2kh krb5kdc[15598](info): set up 2 sockets
Sep 29 18:56:59 dist-test-slave-q2kh krb5kdc[15598](info): commencing operation
krb5kdc: starting...
W20260929 18:57:00.749925 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:00 dist-test-slave-q2kh krb5kdc[15598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:00.873176 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:43043
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:00.953351 15611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:00.953586 15611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:00.953622 15611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:00.954797 15611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:00.954860 15611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:00.954885 15611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:00.954905 15611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:00.954936 15611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:00.956387 15611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:43043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:00.956660 15611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:00.956882 15611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:00.961607 15617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:00.961627 15619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:00.961678 15611 server_base.cc:1034] running on GCE node
W20260929 18:57:00.961621 15616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:00.962106 15611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:00.962419 15611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:00.963578 15611 hybrid_clock.cc:648] HybridClock initialized: now 1790708220963557 us; error 33 us; skew 500 ppm
Sep 29 18:57:00 dist-test-slave-q2kh krb5kdc[15598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708220, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:00.965884 15611 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:00.966341 15611 webserver.cc:460] Webserver started at http://127.29.71.190:37933/ using document root <none> and password file <none>
I20260929 18:57:00.966579 15611 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:00.966647 15611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:00.966779 15611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:00.967847 15611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "1bc6b86564c2496ba2ab717aff3b1b25"
format_stamp: "Formatted at 2026-09-29 18:57:00 on dist-test-slave-q2kh"
server_key: "2a4e9a26fee082a1361fb92df31a3bf2"
server_key_iv: "9ae4cff3bac5072076b662e239c2317d"
server_key_version: "encryptionkey@0"
I20260929 18:57:00.968197 15611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "1bc6b86564c2496ba2ab717aff3b1b25"
format_stamp: "Formatted at 2026-09-29 18:57:00 on dist-test-slave-q2kh"
server_key: "2a4e9a26fee082a1361fb92df31a3bf2"
server_key_iv: "9ae4cff3bac5072076b662e239c2317d"
server_key_version: "encryptionkey@0"
I20260929 18:57:00.969439 15611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:57:00.970100 15626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:00.970273 15611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:00.970371 15611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "1bc6b86564c2496ba2ab717aff3b1b25"
format_stamp: "Formatted at 2026-09-29 18:57:00 on dist-test-slave-q2kh"
server_key: "2a4e9a26fee082a1361fb92df31a3bf2"
server_key_iv: "9ae4cff3bac5072076b662e239c2317d"
server_key_version: "encryptionkey@0"
I20260929 18:57:00.970453 15611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:00.979508 15611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:00.981691 15611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:00.981868 15611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:00.985808 15678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:43043 every 8 connection(s)
I20260929 18:57:00.985811 15611 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:43043
I20260929 18:57:00.986318 15611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:00.987200 15679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:00.987735 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15611
I20260929 18:57:00.987835 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:00.988049 29982 external_mini_cluster.cc:1428] Setting key 0064b00cd4caa88b1c359307d93011d8
I20260929 18:57:00.990190 15679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25: Bootstrap starting.
I20260929 18:57:00.990815 15679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:00.991099 15679 log.cc:824] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:00.991777 15679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25: No bootstrap required, opened a new log
I20260929 18:57:00.993556 15679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43043 } }
I20260929 18:57:00.993721 15679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:00.993772 15679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc6b86564c2496ba2ab717aff3b1b25, State: Initialized, Role: FOLLOWER
I20260929 18:57:00.993932 15679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43043 } }
I20260929 18:57:00.994027 15679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:00.994069 15679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:00.994110 15679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 18:57:00 dist-test-slave-q2kh krb5kdc[15598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:00.994812 15679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43043 } }
I20260929 18:57:00.994925 15679 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc6b86564c2496ba2ab717aff3b1b25; no voters:
I20260929 18:57:00.995100 15679 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:00.995137 15684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:00.995333 15684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc6b86564c2496ba2ab717aff3b1b25, State: Running, Role: LEADER
I20260929 18:57:00.995362 15679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:00.995453 15684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43043 } }
I20260929 18:57:00.996520 15685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43043 } } }
I20260929 18:57:00.996613 15685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:00.996801 15699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:00.997291 15699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:00.997815 15686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc6b86564c2496ba2ab717aff3b1b25. Latest consensus state: current_term: 1 leader_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc6b86564c2496ba2ab717aff3b1b25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 43043 } } }
I20260929 18:57:00.997953 15686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:00.998795 15699 catalog_manager.cc:1347] Generated new cluster ID: b2e391bc00d54dcf94796f2afdc3f56a
I20260929 18:57:00.998847 15699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:00.999573 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:00.988584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.29.71.190:43043)
0929 18:57:00.988743 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:00.988750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:00.988781 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:57:00.989175 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:00.989181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:00.989226 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:00.989346 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:00.990254 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:00.991264 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:00.992126 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:00.992541 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:00.992660 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:00.995335 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:00.995350 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:00.995351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:00.995363 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:00.997717 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:00.998177 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:00.998180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:00.998180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:00.998283 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:00.998516 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:00.998517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:00.998517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:00.998807 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:00.998992 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:00.999245 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 18:57:00.999389 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260929 18:57:01.006037 15699 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:01.006795 15699 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:01.018306 15699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25: Generated new TSK 0
I20260929 18:57:01.018580 15699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:57:01.056990 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43043
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:01.161094 15705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:01.161401 15705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:01.161445 15705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:01.162708 15705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:01.162771 15705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:01.162819 15705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:57:01.164276 15705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:01.164537 15705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:01.164784 15705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:01.169899 15711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:01.169899 15713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:01.169955 15710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:01.170058 15705 server_base.cc:1034] running on GCE node
I20260929 18:57:01.170454 15705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:01.170691 15705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:01.171855 15705 hybrid_clock.cc:648] HybridClock initialized: now 1790708221171842 us; error 48 us; skew 500 ppm
Sep 29 18:57:01 dist-test-slave-q2kh krb5kdc[15598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708221, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:01.174541 15705 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:57:01.174913 15705 webserver.cc:460] Webserver started at http://127.29.71.129:44099/ using document root <none> and password file <none>
I20260929 18:57:01.175137 15705 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:01.175203 15705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:01.175355 15705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:01.176386 15705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "0c1a70e8fbd64bd1a29ae99350f3b389"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "828ccfb4c6611b4492dd2bfcd2160ae1"
server_key_iv: "e1a8784e13e2f9ec0119a338f4f064ae"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.176759 15705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "0c1a70e8fbd64bd1a29ae99350f3b389"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "828ccfb4c6611b4492dd2bfcd2160ae1"
server_key_iv: "e1a8784e13e2f9ec0119a338f4f064ae"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.178103 15705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:57:01.178838 15720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.179000 15705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:57:01.179098 15705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "0c1a70e8fbd64bd1a29ae99350f3b389"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "828ccfb4c6611b4492dd2bfcd2160ae1"
server_key_iv: "e1a8784e13e2f9ec0119a338f4f064ae"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.179180 15705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:01.204005 15705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:01.206115 15705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:01.206338 15705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:01.206579 15705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:01.206902 15705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:01.206959 15705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.206995 15705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:01.207022 15705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.212662 15705 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:43641
I20260929 18:57:01.212719 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:43641 every 8 connection(s)
I20260929 18:57:01.213040 15705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:57:01 dist-test-slave-q2kh krb5kdc[15598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708221, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:01.221922 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.213746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:39711 (local address 127.29.71.190:43043)
0929 18:57:01.213818 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.213821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.214031 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:57:01.214669 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.214670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.214680 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.214740 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:01.215633 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.216127 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.217088 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.217280 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.217344 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.219449 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:01.219450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:01.219450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:01.219457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:01.220732 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.221086 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.221086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.221087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.221137 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.221342 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.221342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.221343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.221486 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:01.221538 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.221768 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.221855 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:57:01.222059 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.213708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43043 (local address 127.29.71.129:39711)
0929 18:57:01.213905 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.213941 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.214579 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.214797 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.214800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.215363 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:01.215551 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.215556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.216203 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.216204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.217013 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.217015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.217375 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.217375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.217470 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.217983 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:01.217991 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:01.219365 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:01.220801 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:01.220807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:01.220809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:01.221020 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:01.221193 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:01.221194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:01.221194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:01.221285 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:01.221539 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:01.221543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:01.221652 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.221732 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.221848 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260929 18:57:01.222601 15834 heartbeater.cc:347] Connected to a master server at 127.29.71.190:43043
I20260929 18:57:01.222707 15834 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:01.222895 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15705
I20260929 18:57:01.222915 15834 heartbeater.cc:511] Master 127.29.71.190:43043 requested a full tablet report, sending...
I20260929 18:57:01.223013 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:57:01.223235 29982 external_mini_cluster.cc:1428] Setting key a8a6e59eec4b316eb8f701d6f83c20cb
I20260929 18:57:01.223508 15642 ts_manager.cc:194] Registered new tserver with Master: 0c1a70e8fbd64bd1a29ae99350f3b389 (127.29.71.129:43641)
I20260929 18:57:01.224386 15642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:39711
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:57:01.249488 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43043
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:01.330931 15839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:01.331254 15839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:01.331306 15839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:01.333405 15839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:01.333577 15839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:01.333670 15839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:57:01.336473 15839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:01.336728 15839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:01.336997 15839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:01.343789 15845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:01.343802 15847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:01.343786 15844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:01.344465 15839 server_base.cc:1034] running on GCE node
I20260929 18:57:01.344846 15839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:01.345090 15839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:01.346244 15839 hybrid_clock.cc:648] HybridClock initialized: now 1790708221346211 us; error 30 us; skew 500 ppm
Sep 29 18:57:01 dist-test-slave-q2kh krb5kdc[15598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708221, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:01.349493 15839 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:57:01.350018 15839 webserver.cc:460] Webserver started at http://127.29.71.130:44737/ using document root <none> and password file <none>
I20260929 18:57:01.350299 15839 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:01.350370 15839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:01.350513 15839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:01.351598 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "585c326ac5974e14abd4e1d046e30fa5"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "7c601c81c52b9731b9b9be4cab2ab61b"
server_key_iv: "6213a5a78f5a550b881895ea04d37982"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.352072 15839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "585c326ac5974e14abd4e1d046e30fa5"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "7c601c81c52b9731b9b9be4cab2ab61b"
server_key_iv: "6213a5a78f5a550b881895ea04d37982"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.354099 15839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 18:57:01.355075 15854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.355290 15839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:01.355413 15839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "585c326ac5974e14abd4e1d046e30fa5"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "7c601c81c52b9731b9b9be4cab2ab61b"
server_key_iv: "6213a5a78f5a550b881895ea04d37982"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.355513 15839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:01.374979 15839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:01.377444 15839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:01.377632 15839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:01.377925 15839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:01.378317 15839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:01.378353 15839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.378376 15839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:01.378389 15839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.384472 15839 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:39943
I20260929 18:57:01.384546 15967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:39943 every 8 connection(s)
I20260929 18:57:01.384858 15839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
Sep 29 18:57:01 dist-test-slave-q2kh krb5kdc[15598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708221, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:01.394040 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.385598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:42929 (local address 127.29.71.190:43043)
0929 18:57:01.385663 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.385666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.385900 (+ 234us) server_negotiation.cc:407] Connection header received
0929 18:57:01.386523 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.386524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.386534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.386603 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:01.387528 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.388046 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.389079 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.389277 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.389334 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.391460 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:01.391461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:01.391461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:01.391469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:01.392746 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.393150 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.393151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.393152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.393213 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.393440 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.393440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.393441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.393556 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:01.393615 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.393891 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.393972 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:57:01.394212 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.385589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43043 (local address 127.29.71.130:42929)
0929 18:57:01.385768 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.385804 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.386413 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.386661 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.386664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.387247 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:01.387445 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.387449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.388124 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.388125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.389001 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.389003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.389353 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.389354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.389448 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.389925 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:01.389932 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:01.391378 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:01.392845 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:01.392850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:01.392852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:01.393068 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:01.393286 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:01.393286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:01.393286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:01.393379 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:01.393618 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:01.393622 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:01.393735 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.393836 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.393959 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260929 18:57:01.394411 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15839
I20260929 18:57:01.394529 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:57:01.394870 15968 heartbeater.cc:347] Connected to a master server at 127.29.71.190:43043
I20260929 18:57:01.394886 29982 external_mini_cluster.cc:1428] Setting key 564a36abef01bd1b9393946681009c31
I20260929 18:57:01.394953 15968 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:01.395138 15968 heartbeater.cc:511] Master 127.29.71.190:43043 requested a full tablet report, sending...
I20260929 18:57:01.395521 15642 ts_manager.cc:194] Registered new tserver with Master: 585c326ac5974e14abd4e1d046e30fa5 (127.29.71.130:39943)
I20260929 18:57:01.396040 15642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:42929
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:57:01.420954 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43043
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:01.502738 15973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:01.502960 15973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:01.502997 15973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:01.504190 15973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:01.504262 15973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:01.504308 15973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:57:01.505764 15973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:01.506016 15973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:01.506287 15973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:01.511123 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:01.511130 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:01.511307 15973 server_base.cc:1034] running on GCE node
W20260929 18:57:01.511359 15981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:01.511582 15973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:01.511775 15973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:01.512921 15973 hybrid_clock.cc:648] HybridClock initialized: now 1790708221512913 us; error 32 us; skew 500 ppm
Sep 29 18:57:01 dist-test-slave-q2kh krb5kdc[15598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708221, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:01.515383 15973 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:57:01.515820 15973 webserver.cc:460] Webserver started at http://127.29.71.131:37349/ using document root <none> and password file <none>
I20260929 18:57:01.516013 15973 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:01.516052 15973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:01.516136 15973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:01.517124 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "621dd917639345bab2c164564979e7c9"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "36d5b146e20529f02d9fee759f99bdb0"
server_key_iv: "f819866f933bfe8685bae3445c4e1aed"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.517431 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "621dd917639345bab2c164564979e7c9"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "36d5b146e20529f02d9fee759f99bdb0"
server_key_iv: "f819866f933bfe8685bae3445c4e1aed"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.518999 15973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:01.519696 15988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.519901 15973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:01.519964 15973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "621dd917639345bab2c164564979e7c9"
format_stamp: "Formatted at 2026-09-29 18:57:01 on dist-test-slave-q2kh"
server_key: "36d5b146e20529f02d9fee759f99bdb0"
server_key_iv: "f819866f933bfe8685bae3445c4e1aed"
server_key_version: "encryptionkey@0"
I20260929 18:57:01.520056 15973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:01.534595 15973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:01.536636 15973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:01.536813 15973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:01.537091 15973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:01.537405 15973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:01.537443 15973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.537493 15973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:01.537519 15973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:01.543541 15973 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:39225
I20260929 18:57:01.543601 16101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:39225 every 8 connection(s)
I20260929 18:57:01.543926 15973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:57:01.545785 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 15973
I20260929 18:57:01.545926 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:57:01.546101 29982 external_mini_cluster.cc:1428] Setting key 1cff9b6cc82f03da07b5c45fb5b3979a
Sep 29 18:57:01 dist-test-slave-q2kh krb5kdc[15598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708221, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:01.553247 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.544738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:37601 (local address 127.29.71.190:43043)
0929 18:57:01.544782 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.544784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.545036 (+ 252us) server_negotiation.cc:407] Connection header received
0929 18:57:01.545502 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.545503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.545514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.545565 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:01.546622 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.547239 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.548217 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.548397 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.548446 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.550683 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:01.550684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:01.550685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:01.550691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:01.552126 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.552489 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.552490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.552491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.552539 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.552742 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.552743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.552743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.552844 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:01.552893 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.553123 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.553182 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:01.553467 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.544758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:43043 (local address 127.29.71.131:37601)
0929 18:57:01.544910 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.544945 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.545402 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.545625 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.545629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.546352 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:01.546547 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.546552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.547311 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.547312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.548145 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.548147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.548468 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.548468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.548566 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.549061 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:01.549069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:01.550596 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:01.552209 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:01.552215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:01.552216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:01.552425 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:01.552595 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:01.552596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:01.552596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:01.552688 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:01.552898 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:01.552901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:01.553015 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.553097 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.553236 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260929 18:57:01.554045 16102 heartbeater.cc:347] Connected to a master server at 127.29.71.190:43043
I20260929 18:57:01.554145 16102 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:01.554378 16102 heartbeater.cc:511] Master 127.29.71.190:43043 requested a full tablet report, sending...
I20260929 18:57:01.554857 15642 ts_manager.cc:194] Registered new tserver with Master: 621dd917639345bab2c164564979e7c9 (127.29.71.131:39225)
I20260929 18:57:01.555238 15642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:37601
I20260929 18:57:01.557575 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:57:01.565208 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.559068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.29.71.190:43043)
0929 18:57:01.559109 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.559110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.559164 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:57:01.559248 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.559248 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.559257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.559299 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:01.559925 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.560390 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.561196 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.561379 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.561428 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.562685 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:01.562686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:01.562686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:01.562692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:01.564050 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.564452 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.564453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.564453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.564500 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:01.564725 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:01.564725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:01.564726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:01.564836 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:01.564913 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.565101 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.565150 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:57:01.566891 15642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:57:01.568460 15642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:57:01.575024 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.570815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:39943 (local address 127.0.0.1:41800)
0929 18:57:01.571060 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.571070 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.571126 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.571424 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.571427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.571429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:01.571508 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.571509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.572875 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.572876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.573807 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.573810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.574728 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.574729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.574808 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.574822 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.574897 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.574947 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":51,"thread_start_us":85,"threads_started":1}
I20260929 18:57:01.575067 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.571045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.29.71.130:39943)
0929 18:57:01.571266 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.571269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.571278 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:57:01.571308 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.571309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.571322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.571391 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:01.571592 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.572772 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.573903 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.574643 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.574721 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.574776 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.574926 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.574975 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260929 18:57:01.575285 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.570856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:43641 (local address 127.0.0.1:54912)
0929 18:57:01.571199 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.571218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.571261 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.571774 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.571775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.571777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:01.571825 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.571826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.572728 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.572729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.574014 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.574016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.575056 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.575057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.575127 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.575131 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.575180 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.575222 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260929 18:57:01.575563 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.571076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.29.71.129:43641)
0929 18:57:01.571546 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.571550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.571561 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:01.571598 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.571600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.571615 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.571689 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:01.571935 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.572612 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.574147 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.574903 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.574968 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.575020 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.575437 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.575481 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":73,"threads_started":1}
I20260929 18:57:01.575812 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.570782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:39225 (local address 127.0.0.1:51870)
0929 18:57:01.570972 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.571010 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.571076 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.571591 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.571591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.571593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:01.571637 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.571638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.573192 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.573193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.574445 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.574447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.575570 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.575571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.575635 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.575638 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.575693 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.575734 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":62,"threads_started":1}
I20260929 18:57:01.575990 15902 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd22b3ffea64c068aa8d6be9a516cca (DEFAULT_TABLE table=test-table [id=f092b27956714c6ba7d0bd50e873e999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:01.576124 15768 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd22b3ffea64c068aa8d6be9a516cca (DEFAULT_TABLE table=test-table [id=f092b27956714c6ba7d0bd50e873e999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:01.576185 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.570872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.29.71.131:39225)
0929 18:57:01.571350 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.571354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.571362 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:57:01.571421 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.571423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.571439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.571514 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:01.571711 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.572815 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.574528 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.575421 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.575487 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.575538 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.576069 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.576114 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":77,"threads_started":1}
I20260929 18:57:01.576422 15768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd22b3ffea64c068aa8d6be9a516cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:01.576422 15902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd22b3ffea64c068aa8d6be9a516cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:01.576768 16036 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd22b3ffea64c068aa8d6be9a516cca (DEFAULT_TABLE table=test-table [id=f092b27956714c6ba7d0bd50e873e999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:01.577071 16036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd22b3ffea64c068aa8d6be9a516cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:01.579583 16119 tablet_bootstrap.cc:492] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Bootstrap starting.
I20260929 18:57:01.580032 16121 tablet_bootstrap.cc:492] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: Bootstrap starting.
I20260929 18:57:01.580314 16119 tablet_bootstrap.cc:654] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:01.580629 16121 tablet_bootstrap.cc:654] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:01.580827 16119 log.cc:824] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:01.581182 16121 log.cc:824] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:01.581540 16119 tablet_bootstrap.cc:492] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: No bootstrap required, opened a new log
I20260929 18:57:01.581527 16120 tablet_bootstrap.cc:492] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Bootstrap starting.
I20260929 18:57:01.581662 16119 ts_tablet_manager.cc:1397] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:57:01.582096 16120 tablet_bootstrap.cc:654] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:01.582641 16120 log.cc:824] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:01.583081 16121 tablet_bootstrap.cc:492] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: No bootstrap required, opened a new log
I20260929 18:57:01.583161 16121 ts_tablet_manager.cc:1397] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:57:01.583529 16119 raft_consensus.cc:348] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.583690 16120 tablet_bootstrap.cc:492] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: No bootstrap required, opened a new log
I20260929 18:57:01.583709 16119 raft_consensus.cc:374] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:01.583747 16120 ts_tablet_manager.cc:1397] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:57:01.583743 16119 raft_consensus.cc:729] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621dd917639345bab2c164564979e7c9, State: Initialized, Role: FOLLOWER
I20260929 18:57:01.583891 16119 consensus_queue.cc:260] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.584157 16119 ts_tablet_manager.cc:1428] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:57:01.584309 16102 heartbeater.cc:503] Master 127.29.71.190:43043 was elected leader, sending a full tablet report...
I20260929 18:57:01.584956 16121 raft_consensus.cc:348] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.585129 16121 raft_consensus.cc:374] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:01.585167 16121 raft_consensus.cc:729] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1a70e8fbd64bd1a29ae99350f3b389, State: Initialized, Role: FOLLOWER
I20260929 18:57:01.585287 16121 consensus_queue.cc:260] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.585379 16120 raft_consensus.cc:348] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.585569 16120 raft_consensus.cc:374] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:01.585587 16121 ts_tablet_manager.cc:1428] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 18:57:01.585614 16120 raft_consensus.cc:729] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585c326ac5974e14abd4e1d046e30fa5, State: Initialized, Role: FOLLOWER
I20260929 18:57:01.585649 15834 heartbeater.cc:503] Master 127.29.71.190:43043 was elected leader, sending a full tablet report...
I20260929 18:57:01.585726 16120 consensus_queue.cc:260] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.586007 16120 ts_tablet_manager.cc:1428] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 18:57:01.586035 15968 heartbeater.cc:503] Master 127.29.71.190:43043 was elected leader, sending a full tablet report...
W20260929 18:57:01.635434 15969 tablet.cc:2369] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:57:01.713712 15835 tablet.cc:2369] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:57:01.794505 16103 tablet.cc:2369] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:57:01.819177 16126 raft_consensus.cc:482] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:57:01.819324 16126 raft_consensus.cc:504] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.819798 16126 leader_election.cc:288] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 621dd917639345bab2c164564979e7c9 (127.29.71.131:39225), 585c326ac5974e14abd4e1d046e30fa5 (127.29.71.130:39943)
I20260929 18:57:01.822852 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.820083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:58195 (local address 127.29.71.130:39943)
0929 18:57:01.820129 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.820132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.820223 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:57:01.820268 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.820269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.820280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.820336 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:01.820518 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.821023 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.822057 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.822505 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.822595 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.822605 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.822751 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.822789 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:57:01.822862 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.819988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:39943 (local address 127.29.71.129:58195)
0929 18:57:01.820133 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.820145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.820214 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.820357 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.820358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.820361 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:01.820442 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.820444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.821109 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.821109 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.821960 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.821962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.822594 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.822596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.822666 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.822672 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.822733 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.822778 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":10,"thread_start_us":79,"threads_started":1}
I20260929 18:57:01.822965 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.819996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:39225 (local address 127.29.71.129:34563)
0929 18:57:01.820156 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.820167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:01.820283 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:01.820428 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:01.820429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:01.820430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:01.820518 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.820519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.821166 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.821166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.821960 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:01.821962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.822675 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.822676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:01.822746 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.822750 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:57:01.822818 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:57:01.822858 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260929 18:57:01.822947 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.820086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:34563 (local address 127.29.71.131:39225)
0929 18:57:01.820132 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.820134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.820290 (+ 156us) server_negotiation.cc:407] Connection header received
0929 18:57:01.820344 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.820345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.820355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.820414 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:01.820593 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.821083 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.822060 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.822589 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.822665 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.822674 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.822844 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.822879 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:57:01.823263 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22b3ffea64c068aa8d6be9a516cca" candidate_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621dd917639345bab2c164564979e7c9" is_pre_election: true
I20260929 18:57:01.823279 15922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22b3ffea64c068aa8d6be9a516cca" candidate_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585c326ac5974e14abd4e1d046e30fa5" is_pre_election: true
I20260929 18:57:01.823418 16056 raft_consensus.cc:2454] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c1a70e8fbd64bd1a29ae99350f3b389 in term 0.
I20260929 18:57:01.823434 15922 raft_consensus.cc:2454] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c1a70e8fbd64bd1a29ae99350f3b389 in term 0.
I20260929 18:57:01.823649 15724 leader_election.cc:302] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1a70e8fbd64bd1a29ae99350f3b389, 621dd917639345bab2c164564979e7c9; no voters:
I20260929 18:57:01.823799 16126 raft_consensus.cc:2781] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:57:01.823863 16126 raft_consensus.cc:482] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:57:01.823890 16126 raft_consensus.cc:3037] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:01.824596 16126 raft_consensus.cc:504] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.824738 16126 leader_election.cc:288] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [CANDIDATE]: Term 1 election: Requested vote from peers 621dd917639345bab2c164564979e7c9 (127.29.71.131:39225), 585c326ac5974e14abd4e1d046e30fa5 (127.29.71.130:39943)
I20260929 18:57:01.824910 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22b3ffea64c068aa8d6be9a516cca" candidate_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621dd917639345bab2c164564979e7c9"
I20260929 18:57:01.824921 15922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd22b3ffea64c068aa8d6be9a516cca" candidate_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585c326ac5974e14abd4e1d046e30fa5"
I20260929 18:57:01.825002 15922 raft_consensus.cc:3037] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:01.824995 16056 raft_consensus.cc:3037] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:01.825870 16056 raft_consensus.cc:2454] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c1a70e8fbd64bd1a29ae99350f3b389 in term 1.
I20260929 18:57:01.825870 15922 raft_consensus.cc:2454] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c1a70e8fbd64bd1a29ae99350f3b389 in term 1.
I20260929 18:57:01.826040 15724 leader_election.cc:302] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1a70e8fbd64bd1a29ae99350f3b389, 621dd917639345bab2c164564979e7c9; no voters:
I20260929 18:57:01.826151 16126 raft_consensus.cc:2781] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:01.826349 16126 raft_consensus.cc:686] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 1 LEADER]: Becoming Leader. State: Replica: 0c1a70e8fbd64bd1a29ae99350f3b389, State: Running, Role: LEADER
I20260929 18:57:01.826457 16126 consensus_queue.cc:237] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } }
I20260929 18:57:01.827432 15643 catalog_manager.cc:5495] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1a70e8fbd64bd1a29ae99350f3b389 (127.29.71.129). New cstate: current_term: 1 leader_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c1a70e8fbd64bd1a29ae99350f3b389" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 43641 } health_report { overall_health: HEALTHY } } }
I20260929 18:57:01.858735 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.856245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.29.71.129:43641)
0929 18:57:01.856316 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:01.856318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:01.856329 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:01.856379 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:01.856379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:01.856397 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:01.856487 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:01.856637 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.857115 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.857901 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:01.858143 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:01.858252 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:01.858409 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:01.858503 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:01.858611 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:01.858642 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 18:57:01.858652 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:57:01.860792 15922 raft_consensus.cc:1261] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 1 FOLLOWER]: Refusing update from remote peer 0c1a70e8fbd64bd1a29ae99350f3b389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:01.860792 16056 raft_consensus.cc:1261] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 1 FOLLOWER]: Refusing update from remote peer 0c1a70e8fbd64bd1a29ae99350f3b389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:01.861181 16126 consensus_queue.cc:1035] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "621dd917639345bab2c164564979e7c9" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:01.861377 16130 consensus_queue.cc:1035] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585c326ac5974e14abd4e1d046e30fa5" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:01.863550 16133 mvcc.cc:205] Tried to move back new op lower bound from 7334740876738654208 to 7334740876601978880. Current Snapshot: MvccSnapshot[applied={T|T < 7334740876738654208}]
I20260929 18:57:01.863735 16135 mvcc.cc:205] Tried to move back new op lower bound from 7334740876738654208 to 7334740876601978880. Current Snapshot: MvccSnapshot[applied={T|T < 7334740876738654208}]
I20260929 18:57:01.863816 16134 mvcc.cc:205] Tried to move back new op lower bound from 7334740876738654208 to 7334740876601978880. Current Snapshot: MvccSnapshot[applied={T|T < 7334740876738654208}]
I20260929 18:57:01.866043 15643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:01.866134 15643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:01.867017 15643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bc6b86564c2496ba2ab717aff3b1b25: Sending DeleteTablet for 3 replicas of tablet 9dd22b3ffea64c068aa8d6be9a516cca
I20260929 18:57:01.867393 15902 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd22b3ffea64c068aa8d6be9a516cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:01 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:41800
I20260929 18:57:01.867460 16036 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd22b3ffea64c068aa8d6be9a516cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:01 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:51870
I20260929 18:57:01.867674 16144 tablet_replica.cc:330] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: stopping tablet replica
I20260929 18:57:01.867774 16144 raft_consensus.cc:2229] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:57:01.867671 15768 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd22b3ffea64c068aa8d6be9a516cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:01 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:54912
I20260929 18:57:01.867877 16144 raft_consensus.cc:2258] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:01.867995 16145 tablet_replica.cc:330] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389: stopping tablet replica
I20260929 18:57:01.868124 16143 tablet_replica.cc:330] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: stopping tablet replica
I20260929 18:57:01.868141 16145 raft_consensus.cc:2229] T 9dd22b3ffea64c068aa8d6be9a516cca P 0c1a70e8fbd64bd1a29ae99350f3b389 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:57:01.868217 16143 raft_consensus.cc:2229] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:57:01.868283 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15705
I20260929 18:57:01.868292 16144 ts_tablet_manager.cc:1905] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:57:01.868309 16143 raft_consensus.cc:2258] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:01.869880 16143 ts_tablet_manager.cc:1905] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:57:01.870755 16144 ts_tablet_manager.cc:1918] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:57:01.870827 16144 log.cc:1196] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/9dd22b3ffea64c068aa8d6be9a516cca
I20260929 18:57:01.871107 16144 ts_tablet_manager.cc:1939] T 9dd22b3ffea64c068aa8d6be9a516cca P 621dd917639345bab2c164564979e7c9: Deleting consensus metadata
I20260929 18:57:01.871562 15630 catalog_manager.cc:4841] TS 621dd917639345bab2c164564979e7c9 (127.29.71.131:39225): tablet 9dd22b3ffea64c068aa8d6be9a516cca (table test-table [id=f092b27956714c6ba7d0bd50e873e999]) successfully deleted
I20260929 18:57:01.871806 16143 ts_tablet_manager.cc:1918] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:57:01.871877 16143 log.cc:1196] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/9dd22b3ffea64c068aa8d6be9a516cca
I20260929 18:57:01.872128 16143 ts_tablet_manager.cc:1939] T 9dd22b3ffea64c068aa8d6be9a516cca P 585c326ac5974e14abd4e1d046e30fa5: Deleting consensus metadata
I20260929 18:57:01.872541 15627 catalog_manager.cc:4841] TS 585c326ac5974e14abd4e1d046e30fa5 (127.29.71.130:39943): tablet 9dd22b3ffea64c068aa8d6be9a516cca (table test-table [id=f092b27956714c6ba7d0bd50e873e999]) successfully deleted
W20260929 18:57:01.875108 15628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:43641 (error 108)
I20260929 18:57:01.875279 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15839
I20260929 18:57:01.878479 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:01.878327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:43641 (local address 127.0.0.1:54928)
0929 18:57:01.878394 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:01.878444 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:43641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 18:57:01.878636 15628 catalog_manager.cc:4584] TS 0c1a70e8fbd64bd1a29ae99350f3b389 (127.29.71.129:43641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dd22b3ffea64c068aa8d6be9a516cca: Network error: Client connection negotiation failed: client connection to 127.29.71.129:43641: connect: Connection refused (error 111)
I20260929 18:57:01.880189 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15973
I20260929 18:57:01.884759 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 15611
2026-09-29T18:57:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2183 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T18:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:01Z Disabled control of system clock
I20260929 18:57:01.911805 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:46173
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:46173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:57:01.991233 16150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:01.991448 16150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:01.991469 16150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:01.992558 16150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:01.992594 16150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:01.992606 16150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:01.992619 16150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:01.992631 16150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:01.993947 16150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:46173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:01.994143 16150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:01.994480 16150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:01.999624 16155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:01.999663 16156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:01.999688 16150 server_base.cc:1034] running on GCE node
W20260929 18:57:01.999665 16158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:02.000094 16150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:02.000388 16150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:02.001550 16150 hybrid_clock.cc:648] HybridClock initialized: now 1790708222001537 us; error 29 us; skew 500 ppm
I20260929 18:57:02.008250 16150 webserver.cc:460] Webserver started at http://127.29.71.190:39107/ using document root <none> and password file <none>
I20260929 18:57:02.008558 16150 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:02.008610 16150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:02.008770 16150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:02.009899 16150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "25151bdb22594a2086a6bf3f825811ce"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "290dfaa028c4290c59c0519875a356ce"
server_key_iv: "2365fa96b1036265e687bff7356fd9b5"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.010295 16150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "25151bdb22594a2086a6bf3f825811ce"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "290dfaa028c4290c59c0519875a356ce"
server_key_iv: "2365fa96b1036265e687bff7356fd9b5"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.011656 16150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:02.012444 16164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.012619 16150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:02.012729 16150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "25151bdb22594a2086a6bf3f825811ce"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "290dfaa028c4290c59c0519875a356ce"
server_key_iv: "2365fa96b1036265e687bff7356fd9b5"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.012807 16150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:02.052114 16150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:02.052479 16150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:02.052647 16150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:02.056890 16150 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:46173
I20260929 18:57:02.056969 16216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:46173 every 8 connection(s)
I20260929 18:57:02.057320 16150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:02.058333 16217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:02.060636 16217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce: Bootstrap starting.
I20260929 18:57:02.061231 16217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:02.061532 16217 log.cc:824] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:02.062250 16217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce: No bootstrap required, opened a new log
I20260929 18:57:02.063825 16217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25151bdb22594a2086a6bf3f825811ce" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46173 } }
I20260929 18:57:02.063964 16217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:02.063987 16217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25151bdb22594a2086a6bf3f825811ce, State: Initialized, Role: FOLLOWER
I20260929 18:57:02.064181 16217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25151bdb22594a2086a6bf3f825811ce" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46173 } }
I20260929 18:57:02.064250 16217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:02.064291 16217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:02.064342 16217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:02.064859 16217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25151bdb22594a2086a6bf3f825811ce" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46173 } }
I20260929 18:57:02.064966 16217 leader_election.cc:302] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25151bdb22594a2086a6bf3f825811ce; no voters:
I20260929 18:57:02.065150 16217 leader_election.cc:288] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:02.065222 16220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:02.065407 16220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [term 1 LEADER]: Becoming Leader. State: Replica: 25151bdb22594a2086a6bf3f825811ce, State: Running, Role: LEADER
I20260929 18:57:02.065418 16217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:02.065562 16220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25151bdb22594a2086a6bf3f825811ce" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46173 } }
I20260929 18:57:02.065991 16222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25151bdb22594a2086a6bf3f825811ce. Latest consensus state: current_term: 1 leader_uuid: "25151bdb22594a2086a6bf3f825811ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25151bdb22594a2086a6bf3f825811ce" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46173 } } }
I20260929 18:57:02.066076 16222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:02.066213 16221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25151bdb22594a2086a6bf3f825811ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25151bdb22594a2086a6bf3f825811ce" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46173 } } }
I20260929 18:57:02.066349 16221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:02.066512 16229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:02.066685 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 16150
I20260929 18:57:02.066809 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:02.066996 16229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:02.067021 29982 external_mini_cluster.cc:1428] Setting key 0327d08a02ee032673ea7bb25f897ce4
I20260929 18:57:02.068917 16229 catalog_manager.cc:1347] Generated new cluster ID: ee03703568fb4933b8590894765f09ff
I20260929 18:57:02.068970 16229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:02.072256 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.067651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.29.71.190:46173)
0929 18:57:02.067831 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.067840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.067942 (+ 102us) server_negotiation.cc:407] Connection header received
0929 18:57:02.068111 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.068119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.068189 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.068359 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:02.068547 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.069522 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.070364 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.070778 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.070913 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.071591 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:02.071595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:02.071595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:02.071775 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:02.071806 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:02.071816 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:02.071888 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.071976 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.072055 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260929 18:57:02.085407 16229 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:02.085958 16229 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:02.088923 16229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce: Generated new TSK 0
I20260929 18:57:02.089124 16229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:02.093374 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46173
--builtin_ntp_servers=127.29.71.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:57:02.174594 16241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:02.174816 16241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:02.174850 16241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:02.175992 16241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:02.176052 16241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:02.176092 16241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:57:02.177434 16241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:02.177700 16241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:02.178040 16241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:02.183700 16246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:02.183683 16247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:02.183939 16249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:02.184043 16241 server_base.cc:1034] running on GCE node
I20260929 18:57:02.184238 16241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:02.184528 16241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:02.185675 16241 hybrid_clock.cc:648] HybridClock initialized: now 1790708222185665 us; error 26 us; skew 500 ppm
I20260929 18:57:02.187012 16241 webserver.cc:460] Webserver started at http://127.29.71.129:46109/ using document root <none> and password file <none>
I20260929 18:57:02.187268 16241 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:02.187311 16241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:02.187446 16241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:02.188577 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "87edb2a376e34a1bb923ded39927b2e6"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "68e3b731d042d082a93538e701d94b5e"
server_key_iv: "1cfa9da3e00a10373e702cf2e97797ae"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.188930 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "87edb2a376e34a1bb923ded39927b2e6"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "68e3b731d042d082a93538e701d94b5e"
server_key_iv: "1cfa9da3e00a10373e702cf2e97797ae"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.190294 16241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:02.191098 16255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.191249 16241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:02.191329 16241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "87edb2a376e34a1bb923ded39927b2e6"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "68e3b731d042d082a93538e701d94b5e"
server_key_iv: "1cfa9da3e00a10373e702cf2e97797ae"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.191422 16241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:02.219660 16241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:02.220014 16241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:02.220177 16241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:02.220434 16241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:02.220799 16241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:02.220854 16241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.220889 16241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:02.220916 16241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.226907 16241 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37869
I20260929 18:57:02.226949 16368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37869 every 8 connection(s)
I20260929 18:57:02.227277 16241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:57:02.228609 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 16241
I20260929 18:57:02.228717 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:57:02.228931 29982 external_mini_cluster.cc:1428] Setting key 42c99d1bfa68faa8831f12cd2bf36174
I20260929 18:57:02.229923 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46173
--builtin_ntp_servers=127.29.71.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:57:02.233036 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.228003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:52489 (local address 127.29.71.190:46173)
0929 18:57:02.228067 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.228069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.228295 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:57:02.228894 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.228895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.228910 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.228975 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:02.229889 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.230517 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.231530 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.231764 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.231855 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.232759 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:02.232761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:02.232761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:02.232766 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:02.232781 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:02.232785 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:02.232833 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.232906 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.232958 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:57:02.233285 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.227990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46173 (local address 127.29.71.129:52489)
0929 18:57:02.228169 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.228208 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.228674 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.229051 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.229055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.229071 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:02.229785 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.229791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.230616 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.230617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.231430 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.231432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.231867 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.231868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.232015 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.232609 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:57:02.232610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:02.232639 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:57:02.232663 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:02.232833 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:02.232838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:02.232841 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.232909 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.233022 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260929 18:57:02.233743 16369 heartbeater.cc:347] Connected to a master server at 127.29.71.190:46173
I20260929 18:57:02.233834 16369 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:02.234041 16369 heartbeater.cc:511] Master 127.29.71.190:46173 requested a full tablet report, sending...
I20260929 18:57:02.234582 16181 ts_manager.cc:194] Registered new tserver with Master: 87edb2a376e34a1bb923ded39927b2e6 (127.29.71.129:37869)
I20260929 18:57:02.235224 16181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.129:52489
W20260929 18:57:02.311616 16372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:02.311853 16372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:02.311887 16372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:02.313020 16372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:02.313086 16372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:02.313128 16372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:57:02.314574 16372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:02.314841 16372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:02.315135 16372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:02.320295 16377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:02.320465 16372 server_base.cc:1034] running on GCE node
W20260929 18:57:02.320295 16380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:02.320336 16378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:02.320780 16372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:02.320991 16372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:02.322192 16372 hybrid_clock.cc:648] HybridClock initialized: now 1790708222322174 us; error 28 us; skew 500 ppm
I20260929 18:57:02.323479 16372 webserver.cc:460] Webserver started at http://127.29.71.130:44523/ using document root <none> and password file <none>
I20260929 18:57:02.323777 16372 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:02.323858 16372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:02.324007 16372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:02.325121 16372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "c84c33d5e764485f8bd16e3e851d706f"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "296570f8bb84de0fd0046df8924f667c"
server_key_iv: "986d4e160ca63401ecb3c8b55a6300cb"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.325541 16372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "c84c33d5e764485f8bd16e3e851d706f"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "296570f8bb84de0fd0046df8924f667c"
server_key_iv: "986d4e160ca63401ecb3c8b55a6300cb"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.327103 16372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:57:02.327855 16386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.328017 16372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:02.328114 16372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "c84c33d5e764485f8bd16e3e851d706f"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "296570f8bb84de0fd0046df8924f667c"
server_key_iv: "986d4e160ca63401ecb3c8b55a6300cb"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.328198 16372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:02.354907 16372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:02.355273 16372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:02.355439 16372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:02.355681 16372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:02.356043 16372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:02.356099 16372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.356142 16372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:02.356189 16372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.361991 16372 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:46673
I20260929 18:57:02.362038 16499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:46673 every 8 connection(s)
I20260929 18:57:02.362442 16372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:57:02.364943 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 16372
I20260929 18:57:02.365051 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:57:02.365264 29982 external_mini_cluster.cc:1428] Setting key 034f5ad291aef425fa2e47d2b8654c56
I20260929 18:57:02.366302 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46173
--builtin_ntp_servers=127.29.71.148:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:57:02.368563 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.363181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:58427 (local address 127.29.71.190:46173)
0929 18:57:02.363253 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.363256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.363509 (+ 253us) server_negotiation.cc:407] Connection header received
0929 18:57:02.363997 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.363998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.364016 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.364088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:02.364788 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.365585 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.366959 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.367307 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.367379 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.368300 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:02.368301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:02.368302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:02.368306 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:02.368320 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:02.368324 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:02.368368 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.368482 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.368504 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:57:02.368815 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.363185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46173 (local address 127.29.71.130:58427)
0929 18:57:02.363385 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.363418 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.363886 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.364158 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.364161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.364169 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:02.364678 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.364682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.365657 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.365659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.366882 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.366884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.367389 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.367390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.367566 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.368138 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:57:02.368139 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:02.368179 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:57:02.368210 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:02.368387 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:02.368392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:02.368395 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.368474 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.368585 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260929 18:57:02.369513 16500 heartbeater.cc:347] Connected to a master server at 127.29.71.190:46173
I20260929 18:57:02.369603 16500 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:02.369861 16500 heartbeater.cc:511] Master 127.29.71.190:46173 requested a full tablet report, sending...
I20260929 18:57:02.370316 16181 ts_manager.cc:194] Registered new tserver with Master: c84c33d5e764485f8bd16e3e851d706f (127.29.71.130:46673)
I20260929 18:57:02.370713 16181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.130:58427
W20260929 18:57:02.450493 16503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:02.450726 16503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:02.450760 16503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:02.451911 16503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:02.451979 16503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:02.452024 16503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:57:02.453368 16503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:46173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:02.453620 16503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:02.453908 16503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:02.460661 16508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:02.460592 16509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:02.460834 16503 server_base.cc:1034] running on GCE node
W20260929 18:57:02.460578 16511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:02.461149 16503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:02.461472 16503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:02.462646 16503 hybrid_clock.cc:648] HybridClock initialized: now 1790708222462624 us; error 33 us; skew 500 ppm
I20260929 18:57:02.464314 16503 webserver.cc:460] Webserver started at http://127.29.71.131:34409/ using document root <none> and password file <none>
I20260929 18:57:02.464614 16503 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:02.464684 16503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:02.464850 16503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:02.466529 16503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "2682a4c4a8e54771a846e4227fa2fd59"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "700e578cf4bafb563f64782ce9e199f7"
server_key_iv: "dcb6cf4998390d135db03fec8afa0ffa"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.467002 16503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "2682a4c4a8e54771a846e4227fa2fd59"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "700e578cf4bafb563f64782ce9e199f7"
server_key_iv: "dcb6cf4998390d135db03fec8afa0ffa"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.468811 16503 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 18:57:02.469735 16517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.469956 16503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:02.470064 16503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "2682a4c4a8e54771a846e4227fa2fd59"
format_stamp: "Formatted at 2026-09-29 18:57:02 on dist-test-slave-q2kh"
server_key: "700e578cf4bafb563f64782ce9e199f7"
server_key_iv: "dcb6cf4998390d135db03fec8afa0ffa"
server_key_version: "encryptionkey@0"
I20260929 18:57:02.470147 16503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:02.485003 16503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:02.485260 16503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:02.485358 16503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:02.485535 16503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:02.485803 16503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:02.485834 16503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.485857 16503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:02.485868 16503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:02.491791 16503 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:38697
I20260929 18:57:02.491822 16630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:38697 every 8 connection(s)
I20260929 18:57:02.492177 16503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:57:02.497699 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.493014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:59717 (local address 127.29.71.190:46173)
0929 18:57:02.493060 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.493062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.493343 (+ 281us) server_negotiation.cc:407] Connection header received
0929 18:57:02.493814 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.493815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.493833 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.493898 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:02.494700 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.495201 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.496205 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.496398 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.496479 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.497455 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:02.497456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:02.497456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:02.497461 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:02.497475 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:02.497479 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:02.497526 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.497609 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.497637 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:57:02.497910 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.492945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46173 (local address 127.29.71.131:59717)
0929 18:57:02.493132 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.493174 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.493717 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.493957 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.493960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.493969 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:02.494614 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.494621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.495315 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.495316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.496116 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.496118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.496519 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.496520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.496644 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.497306 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:57:02.497306 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:02.497344 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:57:02.497369 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:02.497541 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:02.497546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:02.497548 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.497602 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.497684 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260929 18:57:02.498479 16631 heartbeater.cc:347] Connected to a master server at 127.29.71.190:46173
I20260929 18:57:02.498585 16631 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:02.498772 16631 heartbeater.cc:511] Master 127.29.71.190:46173 requested a full tablet report, sending...
I20260929 18:57:02.499223 16181 ts_manager.cc:194] Registered new tserver with Master: 2682a4c4a8e54771a846e4227fa2fd59 (127.29.71.131:38697)
I20260929 18:57:02.499614 16181 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.131:59717
I20260929 18:57:02.502028 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 16503
I20260929 18:57:02.502118 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:57:02.502348 29982 external_mini_cluster.cc:1428] Setting key 5a247da6de90d17c154e5206c3cbb3dd
I20260929 18:57:02.503249 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:57:02.507799 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.504640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.29.71.190:46173)
0929 18:57:02.504718 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.504721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.504806 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:57:02.504882 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.504882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.504895 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.504950 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:02.505117 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.505604 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.506450 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.506642 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.506755 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.507425 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:02.507426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:02.507426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:02.507431 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:02.507445 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:02.507449 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:02.507567 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.507657 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.507670 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:57:02.509250 16181 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:57:02.510917 16181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:57:02.517198 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.513373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.29.71.131:38697)
0929 18:57:02.513521 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.513526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.513536 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:02.513569 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.513571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.513586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.513683 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:02.513842 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.515063 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.516078 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.516809 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.516901 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.516991 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.517077 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.517113 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260929 18:57:02.517186 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.513155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:38697 (local address 127.0.0.1:44754)
0929 18:57:02.513404 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.513416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.513461 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.513693 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.513695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.513698 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:02.513754 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.513756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.515175 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.515176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.515994 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.515997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.516900 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.516901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.516975 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.516989 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.517065 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.517115 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":9,"thread_start_us":88,"threads_started":1}
I20260929 18:57:02.517304 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.512984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37869 (local address 127.0.0.1:60576)
0929 18:57:02.513285 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.513296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.513348 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.513566 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.513569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.513571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:02.513664 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.513666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.514767 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.514768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.516031 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.516034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.517073 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.517074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.517137 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.517141 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.517171 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.517209 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":54,"threads_started":1}
I20260929 18:57:02.517508 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.512948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46673 (local address 127.0.0.1:51504)
0929 18:57:02.513120 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.513167 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.513233 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.513839 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.513839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.513841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:02.513871 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.513873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.515119 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.515120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.516225 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.516227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.517278 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.517279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.517348 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.517351 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.517406 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.517454 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260929 18:57:02.517578 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.513039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.29.71.130:46673)
0929 18:57:02.513580 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.513584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.513593 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:57:02.513632 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.513634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.513673 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.513752 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:02.513952 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.515042 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.516321 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.517053 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.517123 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.517245 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.517440 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.517486 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":92,"threads_started":1}
I20260929 18:57:02.517994 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.513095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.29.71.129:37869)
0929 18:57:02.513317 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.513323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.513343 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:57:02.513448 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.513451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.513468 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.513552 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:02.513797 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.514678 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.516215 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.516966 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.517469 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.517576 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.517604 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.517905 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260929 18:57:02.518503 16434 tablet_service.cc:1467] Processing CreateTablet for tablet 76d18a84b39b40f28343fd7d86414932 (DEFAULT_TABLE table=test-table [id=d8f1f7664ecf4a999945592b47139777]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:02.518687 16303 tablet_service.cc:1467] Processing CreateTablet for tablet 76d18a84b39b40f28343fd7d86414932 (DEFAULT_TABLE table=test-table [id=d8f1f7664ecf4a999945592b47139777]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:02.518944 16434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d18a84b39b40f28343fd7d86414932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:02.519102 16303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d18a84b39b40f28343fd7d86414932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:02.521533 16646 tablet_bootstrap.cc:492] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6: Bootstrap starting.
I20260929 18:57:02.521934 16645 tablet_bootstrap.cc:492] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: Bootstrap starting.
I20260929 18:57:02.521900 16565 tablet_service.cc:1467] Processing CreateTablet for tablet 76d18a84b39b40f28343fd7d86414932 (DEFAULT_TABLE table=test-table [id=d8f1f7664ecf4a999945592b47139777]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:02.522274 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d18a84b39b40f28343fd7d86414932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:02.522292 16646 tablet_bootstrap.cc:654] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:02.522591 16645 tablet_bootstrap.cc:654] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:02.522603 16646 log.cc:824] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:02.523195 16645 log.cc:824] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:02.523370 16646 tablet_bootstrap.cc:492] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6: No bootstrap required, opened a new log
I20260929 18:57:02.523440 16646 ts_tablet_manager.cc:1397] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:57:02.523993 16645 tablet_bootstrap.cc:492] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: No bootstrap required, opened a new log
I20260929 18:57:02.524067 16645 ts_tablet_manager.cc:1397] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:57:02.525682 16649 tablet_bootstrap.cc:492] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Bootstrap starting.
I20260929 18:57:02.525604 16646 raft_consensus.cc:348] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.525774 16646 raft_consensus.cc:374] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:02.525812 16646 raft_consensus.cc:729] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87edb2a376e34a1bb923ded39927b2e6, State: Initialized, Role: FOLLOWER
I20260929 18:57:02.525934 16646 consensus_queue.cc:260] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.526225 16646 ts_tablet_manager.cc:1428] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 18:57:02.526291 16369 heartbeater.cc:503] Master 127.29.71.190:46173 was elected leader, sending a full tablet report...
I20260929 18:57:02.526412 16645 raft_consensus.cc:348] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.526566 16645 raft_consensus.cc:374] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:02.526608 16645 raft_consensus.cc:729] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84c33d5e764485f8bd16e3e851d706f, State: Initialized, Role: FOLLOWER
I20260929 18:57:02.526831 16649 tablet_bootstrap.cc:654] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:02.527130 16649 log.cc:824] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:02.527097 16645 consensus_queue.cc:260] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.527395 16645 ts_tablet_manager.cc:1428] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:57:02.527520 16500 heartbeater.cc:503] Master 127.29.71.190:46173 was elected leader, sending a full tablet report...
I20260929 18:57:02.527887 16649 tablet_bootstrap.cc:492] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: No bootstrap required, opened a new log
I20260929 18:57:02.527956 16649 ts_tablet_manager.cc:1397] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:57:02.530439 16649 raft_consensus.cc:348] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.530582 16649 raft_consensus.cc:374] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:02.530668 16649 raft_consensus.cc:729] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2682a4c4a8e54771a846e4227fa2fd59, State: Initialized, Role: FOLLOWER
I20260929 18:57:02.530807 16649 consensus_queue.cc:260] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.531075 16649 ts_tablet_manager.cc:1428] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 18:57:02.531260 16631 heartbeater.cc:503] Master 127.29.71.190:46173 was elected leader, sending a full tablet report...
W20260929 18:57:02.613003 16501 tablet.cc:2369] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:57:02.669994 16651 raft_consensus.cc:482] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:57:02.670130 16651 raft_consensus.cc:504] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.670584 16651 leader_election.cc:288] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87edb2a376e34a1bb923ded39927b2e6 (127.29.71.129:37869), 2682a4c4a8e54771a846e4227fa2fd59 (127.29.71.131:38697)
I20260929 18:57:02.674611 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.670855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:56283 (local address 127.29.71.131:38697)
0929 18:57:02.670906 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.670908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.670921 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:57:02.670964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.670965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.670974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.671037 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:02.671243 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.672019 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.673644 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.674256 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.674322 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.674330 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.674497 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.674541 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 18:57:02.674611 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.670910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:36533 (local address 127.29.71.129:37869)
0929 18:57:02.670958 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.670960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.671102 (+ 142us) server_negotiation.cc:407] Connection header received
0929 18:57:02.671134 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.671135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.671146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.671228 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:02.671342 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.672018 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.673644 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.674241 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.674322 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.674330 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.674497 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.674541 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 18:57:02.674639 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.670778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:38697 (local address 127.29.71.130:56283)
0929 18:57:02.670840 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.670853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.670907 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.671068 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.671070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.671071 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:02.671150 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.671152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.672127 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.672128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.673528 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.673531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.674332 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.674333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.674411 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.674416 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.674477 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.674531 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":83}
I20260929 18:57:02.674641 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.670810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37869 (local address 127.29.71.130:36533)
0929 18:57:02.670977 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.670985 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:02.671078 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:02.671209 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:02.671209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:02.671211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:02.671271 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.671272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.672127 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.672128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.673530 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:02.673531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.674332 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.674333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:02.674416 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.674418 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:57:02.674477 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 18:57:02.674530 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":121,"thread_start_us":102,"threads_started":1}
I20260929 18:57:02.675076 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d18a84b39b40f28343fd7d86414932" candidate_uuid: "c84c33d5e764485f8bd16e3e851d706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2682a4c4a8e54771a846e4227fa2fd59" is_pre_election: true
I20260929 18:57:02.675069 16323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d18a84b39b40f28343fd7d86414932" candidate_uuid: "c84c33d5e764485f8bd16e3e851d706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87edb2a376e34a1bb923ded39927b2e6" is_pre_election: true
I20260929 18:57:02.675257 16585 raft_consensus.cc:2454] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84c33d5e764485f8bd16e3e851d706f in term 0.
I20260929 18:57:02.675261 16323 raft_consensus.cc:2454] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84c33d5e764485f8bd16e3e851d706f in term 0.
I20260929 18:57:02.675510 16388 leader_election.cc:302] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2682a4c4a8e54771a846e4227fa2fd59, c84c33d5e764485f8bd16e3e851d706f; no voters:
I20260929 18:57:02.675653 16651 raft_consensus.cc:2781] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:57:02.675719 16651 raft_consensus.cc:482] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:57:02.675747 16651 raft_consensus.cc:3037] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:02.676486 16651 raft_consensus.cc:504] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.676632 16651 leader_election.cc:288] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [CANDIDATE]: Term 1 election: Requested vote from peers 87edb2a376e34a1bb923ded39927b2e6 (127.29.71.129:37869), 2682a4c4a8e54771a846e4227fa2fd59 (127.29.71.131:38697)
I20260929 18:57:02.676803 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d18a84b39b40f28343fd7d86414932" candidate_uuid: "c84c33d5e764485f8bd16e3e851d706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2682a4c4a8e54771a846e4227fa2fd59"
I20260929 18:57:02.676832 16323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d18a84b39b40f28343fd7d86414932" candidate_uuid: "c84c33d5e764485f8bd16e3e851d706f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87edb2a376e34a1bb923ded39927b2e6"
I20260929 18:57:02.676877 16585 raft_consensus.cc:3037] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:02.676918 16323 raft_consensus.cc:3037] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:02.677634 16585 raft_consensus.cc:2454] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84c33d5e764485f8bd16e3e851d706f in term 1.
I20260929 18:57:02.677635 16323 raft_consensus.cc:2454] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84c33d5e764485f8bd16e3e851d706f in term 1.
I20260929 18:57:02.677791 16388 leader_election.cc:302] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2682a4c4a8e54771a846e4227fa2fd59, c84c33d5e764485f8bd16e3e851d706f; no voters:
I20260929 18:57:02.677877 16651 raft_consensus.cc:2781] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:02.678047 16651 raft_consensus.cc:686] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 1 LEADER]: Becoming Leader. State: Replica: c84c33d5e764485f8bd16e3e851d706f, State: Running, Role: LEADER
I20260929 18:57:02.678159 16651 consensus_queue.cc:237] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } }
I20260929 18:57:02.678984 16181 catalog_manager.cc:5495] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f reported cstate change: term changed from 0 to 1, leader changed from <none> to c84c33d5e764485f8bd16e3e851d706f (127.29.71.130). New cstate: current_term: 1 leader_uuid: "c84c33d5e764485f8bd16e3e851d706f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84c33d5e764485f8bd16e3e851d706f" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 46673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 } health_report { overall_health: UNKNOWN } } }
I20260929 18:57:02.700898 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.698543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.29.71.130:46673)
0929 18:57:02.698606 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:02.698612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:02.698622 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:02.698658 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:02.698658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:02.698678 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:02.698757 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:02.698880 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.699364 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.700166 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:02.700361 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:02.700417 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:02.700605 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:02.700697 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:02.700773 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:02.700828 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 18:57:02.700838 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:57:02.703038 16323 raft_consensus.cc:1261] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 1 FOLLOWER]: Refusing update from remote peer c84c33d5e764485f8bd16e3e851d706f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:02.703055 16585 raft_consensus.cc:1261] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 1 FOLLOWER]: Refusing update from remote peer c84c33d5e764485f8bd16e3e851d706f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:02.703428 16651 consensus_queue.cc:1035] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [LEADER]: Connected to new peer: Peer: permanent_uuid: "87edb2a376e34a1bb923ded39927b2e6" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:02.703572 16655 consensus_queue.cc:1035] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2682a4c4a8e54771a846e4227fa2fd59" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:02.706094 16658 mvcc.cc:205] Tried to move back new op lower bound from 7334740880188915712 to 7334740880090546176. Current Snapshot: MvccSnapshot[applied={T|T < 7334740880188915712}]
I20260929 18:57:02.706471 16659 mvcc.cc:205] Tried to move back new op lower bound from 7334740880188915712 to 7334740880090546176. Current Snapshot: MvccSnapshot[applied={T|T < 7334740880188915712}]
I20260929 18:57:02.707818 16661 mvcc.cc:205] Tried to move back new op lower bound from 7334740880188915712 to 7334740880090546176. Current Snapshot: MvccSnapshot[applied={T|T < 7334740880188915712 or (T in {7334740880188915712})}]
I20260929 18:57:02.708429 16181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:02.708526 16181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:02.709316 16181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25151bdb22594a2086a6bf3f825811ce: Sending DeleteTablet for 3 replicas of tablet 76d18a84b39b40f28343fd7d86414932
I20260929 18:57:02.709697 16303 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d18a84b39b40f28343fd7d86414932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:02 UTC) from {username='slave'} at 127.0.0.1:60576
I20260929 18:57:02.709702 16565 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d18a84b39b40f28343fd7d86414932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:02 UTC) from {username='slave'} at 127.0.0.1:44754
I20260929 18:57:02.709664 16434 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d18a84b39b40f28343fd7d86414932 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:02 UTC) from {username='slave'} at 127.0.0.1:51504
I20260929 18:57:02.709944 16669 tablet_replica.cc:330] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f: stopping tablet replica
I20260929 18:57:02.710039 16669 raft_consensus.cc:2229] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:57:02.710032 16667 tablet_replica.cc:330] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: stopping tablet replica
I20260929 18:57:02.710194 16667 raft_consensus.cc:2229] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:57:02.710326 16667 raft_consensus.cc:2258] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:02.710321 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 16241
I20260929 18:57:02.710386 16668 tablet_replica.cc:330] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6: stopping tablet replica
I20260929 18:57:02.710476 16668 raft_consensus.cc:2229] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:57:02.710557 16668 raft_consensus.cc:2258] T 76d18a84b39b40f28343fd7d86414932 P 87edb2a376e34a1bb923ded39927b2e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:02.711331 16667 ts_tablet_manager.cc:1905] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:57:02.712888 16667 ts_tablet_manager.cc:1918] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:57:02.712985 16667 log.cc:1196] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/76d18a84b39b40f28343fd7d86414932
I20260929 18:57:02.713235 16667 ts_tablet_manager.cc:1939] T 76d18a84b39b40f28343fd7d86414932 P 2682a4c4a8e54771a846e4227fa2fd59: Deleting consensus metadata
I20260929 18:57:02.713639 16166 catalog_manager.cc:4841] TS 2682a4c4a8e54771a846e4227fa2fd59 (127.29.71.131:38697): tablet 76d18a84b39b40f28343fd7d86414932 (table test-table [id=d8f1f7664ecf4a999945592b47139777]) successfully deleted
W20260929 18:57:02.718250 16165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:37869 (error 108)
I20260929 18:57:02.718386 16669 raft_consensus.cc:2258] T 76d18a84b39b40f28343fd7d86414932 P c84c33d5e764485f8bd16e3e851d706f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:02.718573 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 16372
I20260929 18:57:02.719237 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.718750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37869 (local address 127.0.0.1:60584)
0929 18:57:02.719145 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.719185 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373}
W20260929 18:57:02.719379 16165 catalog_manager.cc:4584] TS 87edb2a376e34a1bb923ded39927b2e6 (127.29.71.129:37869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76d18a84b39b40f28343fd7d86414932: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37869: connect: Connection refused (error 111)
I20260929 18:57:02.724825 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 16503
I20260929 18:57:02.724915 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:02.724778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:46673 (local address 127.0.0.1:51514)
0929 18:57:02.724861 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:02.724888 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.130:46673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:57:02.729570 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 16150
2026-09-29T18:57:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (844 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:02 dist-test-slave-q2kh krb5kdc[16673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:02 dist-test-slave-q2kh krb5kdc[16673](info): set up 2 sockets
Sep 29 18:57:02 dist-test-slave-q2kh krb5kdc[16673](info): commencing operation
krb5kdc: starting...
W20260929 18:57:03.762039 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:03 dist-test-slave-q2kh krb5kdc[16673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:03.881995 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42287
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:42287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:03.963004 16686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:03.963205 16686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:03.963227 16686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:03.964360 16686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:03.964402 16686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:03.964416 16686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:03.964429 16686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:03.964481 16686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:03.965899 16686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:42287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:42287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:03.966142 16686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:03.966403 16686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:03.971081 16694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:03.971081 16691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:03.971148 16692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:03.971421 16686 server_base.cc:1034] running on GCE node
I20260929 18:57:03.971653 16686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:03.971977 16686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:03.973121 16686 hybrid_clock.cc:648] HybridClock initialized: now 1790708223973104 us; error 37 us; skew 500 ppm
Sep 29 18:57:03 dist-test-slave-q2kh krb5kdc[16673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708223, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:03.975620 16686 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:03.976140 16686 webserver.cc:460] Webserver started at http://127.29.71.190:40091/ using document root <none> and password file <none>
I20260929 18:57:03.976378 16686 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:03.976456 16686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:03.976613 16686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:03.977660 16686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "19c24d8b9dcf4370ab077f48cc2afe6d"
format_stamp: "Formatted at 2026-09-29 18:57:03 on dist-test-slave-q2kh"
server_key: "4a1d6211ad3be44674245f6ce5f3a278"
server_key_iv: "b2ff1ccbc158547ed90859502323512e"
server_key_version: "encryptionkey@0"
I20260929 18:57:03.978034 16686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "19c24d8b9dcf4370ab077f48cc2afe6d"
format_stamp: "Formatted at 2026-09-29 18:57:03 on dist-test-slave-q2kh"
server_key: "4a1d6211ad3be44674245f6ce5f3a278"
server_key_iv: "b2ff1ccbc158547ed90859502323512e"
server_key_version: "encryptionkey@0"
I20260929 18:57:03.979405 16686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:03.980154 16701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:03.980331 16686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:03.980432 16686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "19c24d8b9dcf4370ab077f48cc2afe6d"
format_stamp: "Formatted at 2026-09-29 18:57:03 on dist-test-slave-q2kh"
server_key: "4a1d6211ad3be44674245f6ce5f3a278"
server_key_iv: "b2ff1ccbc158547ed90859502323512e"
server_key_version: "encryptionkey@0"
I20260929 18:57:03.980520 16686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:03.994994 16686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:03.996896 16686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:03.997054 16686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:04.001307 16686 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:42287
I20260929 18:57:04.001339 16753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:42287 every 8 connection(s)
I20260929 18:57:04.001713 16686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:04.002667 16754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:04.004877 16754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d: Bootstrap starting.
I20260929 18:57:04.005393 16754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:04.005625 16754 log.cc:824] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:04.006336 16754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d: No bootstrap required, opened a new log
I20260929 18:57:04.006830 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 16686
I20260929 18:57:04.006955 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:04.007138 29982 external_mini_cluster.cc:1428] Setting key 6037483b8711ce6c5e0e7546cfd98852
I20260929 18:57:04.009011 16754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42287 } }
I20260929 18:57:04.009182 16754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:04.009219 16754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c24d8b9dcf4370ab077f48cc2afe6d, State: Initialized, Role: FOLLOWER
I20260929 18:57:04.009352 16754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42287 } }
I20260929 18:57:04.009434 16754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:04.009456 16754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:04.009526 16754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:04.010182 16754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42287 } }
I20260929 18:57:04.010288 16754 leader_election.cc:302] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c24d8b9dcf4370ab077f48cc2afe6d; no voters:
I20260929 18:57:04.010435 16754 leader_election.cc:288] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:04.010491 16759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:04.010649 16759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [term 1 LEADER]: Becoming Leader. State: Replica: 19c24d8b9dcf4370ab077f48cc2afe6d, State: Running, Role: LEADER
I20260929 18:57:04.010704 16754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:04.010735 16759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42287 } }
I20260929 18:57:04.011173 16760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42287 } } }
I20260929 18:57:04.011288 16760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:04.011168 16761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c24d8b9dcf4370ab077f48cc2afe6d. Latest consensus state: current_term: 1 leader_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c24d8b9dcf4370ab077f48cc2afe6d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 42287 } } }
I20260929 18:57:04.011382 16761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d [sys.catalog]: This master's current role is: LEADER
W20260929 18:57:04.012581 16774 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:04.012639 16774 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:04.012693 16775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:04.012838 16775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:04.014488 16775 catalog_manager.cc:1347] Generated new cluster ID: 790cd2e01d214b52bf4f775977d263d7
I20260929 18:57:04.014551 16775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:57:04 dist-test-slave-q2kh krb5kdc[16673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:04.019585 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.007685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.29.71.190:42287)
0929 18:57:04.007889 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.007898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.007934 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:57:04.008959 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.008969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.009034 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.009187 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:04.009953 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.011419 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.012508 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.013074 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.013202 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.015888 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:04.015892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:04.015892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:04.015904 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:04.018062 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.018479 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.018482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.018482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.018571 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.018789 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.018790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.018790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.019020 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:04.019220 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.019297 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.019433 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260929 18:57:04.057049 16775 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:04.057667 16775 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:04.061314 16775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d: Generated new TSK 0
I20260929 18:57:04.061501 16775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.71.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
Entry for principal kudu/127.29.71.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab.
I20260929 18:57:04.095216 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42287
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:04.194721 16780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:04.195065 16780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:04.195093 16780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:04.196447 16780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:04.196499 16780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:04.196528 16780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:57:04.198030 16780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.129.keytab
--principal=kudu/127.29.71.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:04.198302 16780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:04.198563 16780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:04.203799 16788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.203792 16785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.203822 16786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:04.204260 16780 server_base.cc:1034] running on GCE node
I20260929 18:57:04.204521 16780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:04.204763 16780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:04.205930 16780 hybrid_clock.cc:648] HybridClock initialized: now 1790708224205914 us; error 26 us; skew 500 ppm
Sep 29 18:57:04 dist-test-slave-q2kh krb5kdc[16673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708224, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:04.208447 16780 init.cc:377] Logged in from keytab as kudu/127.29.71.129@KRBTEST.COM (short username kudu)
I20260929 18:57:04.208922 16780 webserver.cc:460] Webserver started at http://127.29.71.129:35525/ using document root <none> and password file <none>
I20260929 18:57:04.209160 16780 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:04.209228 16780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:04.209368 16780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:04.210507 16780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "d92a548a2eb44f1c95dea364a61da793"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "cf57b24861299bfed6f6496d8f0d5981"
server_key_iv: "f38b3d84b85cdef3d623f1c0443954f5"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.210899 16780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "d92a548a2eb44f1c95dea364a61da793"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "cf57b24861299bfed6f6496d8f0d5981"
server_key_iv: "f38b3d84b85cdef3d623f1c0443954f5"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.212532 16780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:04.213295 16795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.213480 16780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:04.213577 16780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "d92a548a2eb44f1c95dea364a61da793"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "cf57b24861299bfed6f6496d8f0d5981"
server_key_iv: "f38b3d84b85cdef3d623f1c0443954f5"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.213663 16780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:04.231575 16780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:04.233703 16780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:04.233886 16780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:04.234131 16780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:04.234536 16780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:04.234593 16780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.234633 16780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:04.234661 16780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.241214 16780 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:36811
I20260929 18:57:04.241344 16908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:36811 every 8 connection(s)
I20260929 18:57:04.241643 16780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:57:04 dist-test-slave-q2kh krb5kdc[16673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708224, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.129@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:04.250675 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 16780
I20260929 18:57:04.250767 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:57:04.250965 29982 external_mini_cluster.cc:1428] Setting key e57d98624b03b1d4fcdc6347a52773ab
I20260929 18:57:04.251376 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.242450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:50251 (local address 127.29.71.190:42287)
0929 18:57:04.242521 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.242523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.242751 (+ 228us) server_negotiation.cc:407] Connection header received
0929 18:57:04.243290 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.243291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.243304 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.243362 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:04.244358 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.244888 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.245911 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.246139 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.246255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.248435 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:04.248437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:04.248437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:04.248444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:04.249685 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.250268 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.250269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.250270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.250333 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.250590 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.250591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.250591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.250713 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:04.250847 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.251188 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.251283 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:57:04.251591 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.242435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42287 (local address 127.29.71.129:50251)
0929 18:57:04.242623 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.242659 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.243181 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.243436 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.243440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.244030 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:04.244245 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.244250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.244988 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.244990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.245802 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.245804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.246267 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.246268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.246400 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.246913 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:04.246922 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:04.248329 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:04.249885 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:04.249892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:04.249895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:04.250117 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:04.250404 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:04.250405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:04.250406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:04.250517 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:04.250875 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:04.250882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:04.251035 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.251166 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.251340 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260929 18:57:04.252203 16909 heartbeater.cc:347] Connected to a master server at 127.29.71.190:42287
I20260929 18:57:04.252295 16909 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:04.252496 16909 heartbeater.cc:511] Master 127.29.71.190:42287 requested a full tablet report, sending...
I20260929 18:57:04.253051 16718 ts_manager.cc:194] Registered new tserver with Master: d92a548a2eb44f1c95dea364a61da793 (127.29.71.129:36811)
I20260929 18:57:04.253710 16718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.129@KRBTEST.COM'} at 127.29.71.129:50251
WARNING: no policy specified for kudu/127.29.71.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
Entry for principal kudu/127.29.71.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab.
I20260929 18:57:04.277985 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42287
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:04.358057 16914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:04.358299 16914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:04.358333 16914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:04.359457 16914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:04.359521 16914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:04.359566 16914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:57:04.360992 16914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.130.keytab
--principal=kudu/127.29.71.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:04.361248 16914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:04.361518 16914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:04.366503 16920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.366479 16922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.366582 16919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:04.366750 16914 server_base.cc:1034] running on GCE node
I20260929 18:57:04.367027 16914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:04.367267 16914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:04.368430 16914 hybrid_clock.cc:648] HybridClock initialized: now 1790708224368411 us; error 29 us; skew 500 ppm
Sep 29 18:57:04 dist-test-slave-q2kh krb5kdc[16673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708224, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:04.370987 16914 init.cc:377] Logged in from keytab as kudu/127.29.71.130@KRBTEST.COM (short username kudu)
I20260929 18:57:04.371447 16914 webserver.cc:460] Webserver started at http://127.29.71.130:44541/ using document root <none> and password file <none>
I20260929 18:57:04.371668 16914 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:04.371734 16914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:04.371886 16914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:04.372956 16914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "7e9af8b3fa1540d98b16ebab8fea690e"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "fc894ad15423b71d85eca7e4e0e6cc91"
server_key_iv: "ed871fffcfe19ed0f1889a5645908e19"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.373344 16914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "7e9af8b3fa1540d98b16ebab8fea690e"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "fc894ad15423b71d85eca7e4e0e6cc91"
server_key_iv: "ed871fffcfe19ed0f1889a5645908e19"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.374696 16914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:57:04.375418 16929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.375581 16914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:04.375676 16914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "7e9af8b3fa1540d98b16ebab8fea690e"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "fc894ad15423b71d85eca7e4e0e6cc91"
server_key_iv: "ed871fffcfe19ed0f1889a5645908e19"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.375756 16914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:04.420324 16914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:04.422865 16914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:04.423062 16914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:04.423337 16914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:04.423708 16914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:04.423769 16914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.423815 16914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:04.423861 16914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.430253 16914 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:35121
I20260929 18:57:04.430302 17042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:35121 every 8 connection(s)
I20260929 18:57:04.430734 16914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:57:04.433353 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 16914
I20260929 18:57:04.433467 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:57:04.433637 29982 external_mini_cluster.cc:1428] Setting key d6a360fb7e099d37afc68dcecacce6bb
Sep 29 18:57:04 dist-test-slave-q2kh krb5kdc[16673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708224, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.130@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:04.442745 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.431646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:54717 (local address 127.29.71.190:42287)
0929 18:57:04.431772 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.431775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.431985 (+ 210us) server_negotiation.cc:407] Connection header received
0929 18:57:04.432516 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.432517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.432529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.432582 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:04.433836 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.434556 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.436141 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.436504 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.436596 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.439262 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:04.439265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:04.439265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:04.439274 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:04.441031 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.441498 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.441499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.441499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.441563 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.441855 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.441856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.441856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.442006 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:04.442088 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.442544 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.442635 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:57:04.442863 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.431551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42287 (local address 127.29.71.130:54717)
0929 18:57:04.431851 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.431893 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.432411 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.432673 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.432677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.433357 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:04.433660 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.433668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.434659 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.434660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.436040 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.436042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.436605 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.436606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.436710 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.437342 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:04.437351 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:04.439157 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:04.441138 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:04.441145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:04.441147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:04.441411 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:04.441657 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:04.441659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:04.441659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:04.441772 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:04.442126 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:04.442143 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:04.442389 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.442488 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.442616 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.29.71.131@KRBTEST.COM; defaulting to no policy
I20260929 18:57:04.443444 17043 heartbeater.cc:347] Connected to a master server at 127.29.71.190:42287
I20260929 18:57:04.443547 17043 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:04.443694 17043 heartbeater.cc:511] Master 127.29.71.190:42287 requested a full tablet report, sending...
I20260929 18:57:04.444108 16718 ts_manager.cc:194] Registered new tserver with Master: 7e9af8b3fa1540d98b16ebab8fea690e (127.29.71.130:35121)
I20260929 18:57:04.444499 16718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.130@KRBTEST.COM'} at 127.29.71.130:54717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
Entry for principal kudu/127.29.71.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab.
I20260929 18:57:04.461807 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42287
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:04.543992 17048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:04.544193 17048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:04.544217 17048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:04.545286 17048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:04.545318 17048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:04.545344 17048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:57:04.546867 17048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.131.keytab
--principal=kudu/127.29.71.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:42287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:04.547045 17048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:04.547207 17048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:04.552076 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.552066 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.552052 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:04.552302 17048 server_base.cc:1034] running on GCE node
I20260929 18:57:04.552522 17048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:04.552760 17048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:04.553946 17048 hybrid_clock.cc:648] HybridClock initialized: now 1790708224553931 us; error 29 us; skew 500 ppm
Sep 29 18:57:04 dist-test-slave-q2kh krb5kdc[16673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708224, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:04.556560 17048 init.cc:377] Logged in from keytab as kudu/127.29.71.131@KRBTEST.COM (short username kudu)
I20260929 18:57:04.556938 17048 webserver.cc:460] Webserver started at http://127.29.71.131:35149/ using document root <none> and password file <none>
I20260929 18:57:04.557119 17048 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:04.557158 17048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:04.557245 17048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:04.558331 17048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "d34f06210c444357ab5e4c2603ca26ff"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "49a3e23fe7410c40f887bd33c9aa6b8d"
server_key_iv: "aad55f18aec84fe00676f27e95243c6e"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.558625 17048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "d34f06210c444357ab5e4c2603ca26ff"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "49a3e23fe7410c40f887bd33c9aa6b8d"
server_key_iv: "aad55f18aec84fe00676f27e95243c6e"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.559829 17048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:04.560501 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.560674 17048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:57:04.560755 17048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "d34f06210c444357ab5e4c2603ca26ff"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "49a3e23fe7410c40f887bd33c9aa6b8d"
server_key_iv: "aad55f18aec84fe00676f27e95243c6e"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.560814 17048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:04.571534 17048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:04.573364 17048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:04.573460 17048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:04.573617 17048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:04.573856 17048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:04.573886 17048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.573907 17048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:04.573920 17048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.579823 17048 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:40833
I20260929 18:57:04.579891 17176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:40833 every 8 connection(s)
I20260929 18:57:04.580200 17048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
Sep 29 18:57:04 dist-test-slave-q2kh krb5kdc[16673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708224, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.131@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:04.586598 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17048
I20260929 18:57:04.586680 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:57:04.586946 29982 external_mini_cluster.cc:1428] Setting key 6389c815cd6b266ad2ad9719e38041a7
I20260929 18:57:04.589847 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.580937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:41397 (local address 127.29.71.190:42287)
0929 18:57:04.581019 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.581022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.581229 (+ 207us) server_negotiation.cc:407] Connection header received
0929 18:57:04.581783 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.581784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.581797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.581862 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:04.582869 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.583398 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.584396 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.584606 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.584688 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.586965 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:04.586967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:04.586967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:04.586975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:04.588453 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.588867 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.588868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.588869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.588930 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.589197 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.589198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.589198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.589320 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:04.589391 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.589671 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.589747 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:57:04.590026 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.580908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:42287 (local address 127.29.71.131:41397)
0929 18:57:04.581082 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.581123 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.581637 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.581931 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.581936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.582557 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:04.582744 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.582748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.583499 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.583501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.584290 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.584292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.584697 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.584698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.584825 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.585337 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:04.585345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:04.586865 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:04.588545 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:04.588552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:04.588554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:04.588775 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:04.589000 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:04.589000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:04.589001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:04.589117 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:04.589394 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:04.589401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:04.589537 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.589640 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.589795 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260929 18:57:04.590541 17177 heartbeater.cc:347] Connected to a master server at 127.29.71.190:42287
I20260929 18:57:04.590633 17177 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:04.590780 17177 heartbeater.cc:511] Master 127.29.71.190:42287 requested a full tablet report, sending...
I20260929 18:57:04.591133 16718 ts_manager.cc:194] Registered new tserver with Master: d34f06210c444357ab5e4c2603ca26ff (127.29.71.131:40833)
I20260929 18:57:04.591463 16718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.71.131@KRBTEST.COM'} at 127.29.71.131:41397
I20260929 18:57:04.598896 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:57:04.606653 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.600314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.29.71.190:42287)
0929 18:57:04.600365 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.600367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.600394 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:57:04.600470 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.600471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.600481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.600525 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:04.601187 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.601679 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.602505 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.602698 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.602787 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.603993 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:04.603995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:04.603995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:04.604001 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:04.605398 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.605768 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.605770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.605770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.605830 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:04.606026 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:04.606028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:04.606028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:04.606236 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:04.606349 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.606462 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.606549 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:57:04.608494 16718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:57:04.610212 16718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:57:04.617195 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.612811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.29.71.130:35121)
0929 18:57:04.612984 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.612989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.613000 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:04.613039 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.613042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.613060 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.613152 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:04.613527 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.614672 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.616107 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.616696 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.616785 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.616855 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.617070 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.617115 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260929 18:57:04.617187 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.612701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:35121 (local address 127.0.0.1:42830)
0929 18:57:04.612803 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.612876 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.612939 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.613358 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.613361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.613363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:04.613431 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.613433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.614779 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.614780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.616019 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.616022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.616886 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.616887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.616953 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.616963 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.617053 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.617110 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":58,"threads_started":1}
I20260929 18:57:04.617300 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.612902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:40833 (local address 127.0.0.1:53634)
0929 18:57:04.613281 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.613291 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.613349 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.613782 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.613783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.613785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:04.613821 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.613823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.615010 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.615012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.616067 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.616069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.617083 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.617084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.617139 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.617142 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.617189 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.617228 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":67,"threads_started":1}
I20260929 18:57:04.617514 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.613289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.29.71.131:40833)
0929 18:57:04.613567 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.613571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.613581 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:04.613618 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.613620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.613634 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.613712 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:04.613896 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.614916 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.616158 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.616926 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.616993 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.617049 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.617396 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.617441 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":88,"threads_started":1}
I20260929 18:57:04.618652 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.613054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36811 (local address 127.0.0.1:33562)
0929 18:57:04.614279 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.614287 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.614354 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.614535 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.614536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.614538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:04.614585 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.614587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.615850 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.615851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.617018 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.617021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.618405 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.618405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.618466 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.618470 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.618546 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.618588 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"thread_start_us":86,"threads_started":1}
I20260929 18:57:04.618692 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.613191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.29.71.129:36811)
0929 18:57:04.613465 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.613470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.614350 (+ 880us) server_negotiation.cc:407] Connection header received
0929 18:57:04.614426 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.614428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.614442 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.614527 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:04.614642 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.615749 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.617110 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.617865 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.617902 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.617967 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.618567 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.618613 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260929 18:57:04.618927 16977 tablet_service.cc:1467] Processing CreateTablet for tablet f1e58b69c8484d56962531baa20a67e9 (DEFAULT_TABLE table=test-table [id=349e58ee55104c428d65228a74f63e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:04.618925 17111 tablet_service.cc:1467] Processing CreateTablet for tablet f1e58b69c8484d56962531baa20a67e9 (DEFAULT_TABLE table=test-table [id=349e58ee55104c428d65228a74f63e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:04.619289 17111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e58b69c8484d56962531baa20a67e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:04.619289 16977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e58b69c8484d56962531baa20a67e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:04.619325 16843 tablet_service.cc:1467] Processing CreateTablet for tablet f1e58b69c8484d56962531baa20a67e9 (DEFAULT_TABLE table=test-table [id=349e58ee55104c428d65228a74f63e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:04.619683 16843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e58b69c8484d56962531baa20a67e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:04.622232 17194 tablet_bootstrap.cc:492] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793: Bootstrap starting.
I20260929 18:57:04.622659 17195 tablet_bootstrap.cc:492] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Bootstrap starting.
I20260929 18:57:04.622949 17196 tablet_bootstrap.cc:492] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Bootstrap starting.
I20260929 18:57:04.623186 17194 tablet_bootstrap.cc:654] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:04.623356 17195 tablet_bootstrap.cc:654] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:04.623629 17196 tablet_bootstrap.cc:654] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:04.623824 17194 log.cc:824] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:04.623945 17195 log.cc:824] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:04.624238 17196 log.cc:824] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:04.624665 17194 tablet_bootstrap.cc:492] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793: No bootstrap required, opened a new log
I20260929 18:57:04.624683 17195 tablet_bootstrap.cc:492] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: No bootstrap required, opened a new log
I20260929 18:57:04.624742 17195 ts_tablet_manager.cc:1397] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:57:04.624742 17194 ts_tablet_manager.cc:1397] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:57:04.624943 17196 tablet_bootstrap.cc:492] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: No bootstrap required, opened a new log
I20260929 18:57:04.625022 17196 ts_tablet_manager.cc:1397] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:57:04.626543 17196 raft_consensus.cc:348] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.626751 17196 raft_consensus.cc:374] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:04.626799 17196 raft_consensus.cc:729] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9af8b3fa1540d98b16ebab8fea690e, State: Initialized, Role: FOLLOWER
I20260929 18:57:04.626715 17194 raft_consensus.cc:348] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.626705 17195 raft_consensus.cc:348] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.626886 17194 raft_consensus.cc:374] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:04.626886 17195 raft_consensus.cc:374] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:04.626928 17194 raft_consensus.cc:729] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92a548a2eb44f1c95dea364a61da793, State: Initialized, Role: FOLLOWER
I20260929 18:57:04.626928 17195 raft_consensus.cc:729] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34f06210c444357ab5e4c2603ca26ff, State: Initialized, Role: FOLLOWER
I20260929 18:57:04.626935 17196 consensus_queue.cc:260] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.627048 17195 consensus_queue.cc:260] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.627046 17194 consensus_queue.cc:260] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.627228 17196 ts_tablet_manager.cc:1428] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:57:04.627318 17195 ts_tablet_manager.cc:1428] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:57:04.627318 17194 ts_tablet_manager.cc:1428] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:57:04.627373 17043 heartbeater.cc:503] Master 127.29.71.190:42287 was elected leader, sending a full tablet report...
I20260929 18:57:04.627444 16909 heartbeater.cc:503] Master 127.29.71.190:42287 was elected leader, sending a full tablet report...
I20260929 18:57:04.627697 17177 heartbeater.cc:503] Master 127.29.71.190:42287 was elected leader, sending a full tablet report...
I20260929 18:57:04.677438 17202 raft_consensus.cc:482] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:57:04.677579 17202 raft_consensus.cc:504] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.678046 17202 leader_election.cc:288] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e9af8b3fa1540d98b16ebab8fea690e (127.29.71.130:35121), d34f06210c444357ab5e4c2603ca26ff (127.29.71.131:40833)
W20260929 18:57:04.681402 17044 tablet.cc:2369] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:57:04.682246 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.678398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:57949 (local address 127.29.71.130:35121)
0929 18:57:04.678475 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.678478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.678490 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:57:04.678641 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.678642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.678653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.678773 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:04.678916 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.679757 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.681303 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.681823 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.681886 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.681893 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.682108 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.682172 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:57:04.682276 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.678334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:35121 (local address 127.29.71.129:57949)
0929 18:57:04.678400 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.678415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.678502 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.678729 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.678730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.678732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:04.678815 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.678818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.679867 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.679868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.681176 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.681179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.681920 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.681922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.682021 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.682024 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.682081 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.682145 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":78}
I20260929 18:57:04.682299 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.678576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:50361 (local address 127.29.71.131:40833)
0929 18:57:04.678663 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.678666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.678785 (+ 119us) server_negotiation.cc:407] Connection header received
0929 18:57:04.678817 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.678818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.678828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.678889 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:04.679012 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.679797 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.681293 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.681849 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.681957 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.681966 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.682149 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.682211 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:57:04.682509 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.678495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:40833 (local address 127.29.71.129:50361)
0929 18:57:04.678699 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.678709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:04.678767 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:04.678888 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:04.678889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:04.678890 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:04.678945 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.678946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.679874 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.679875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.681176 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:04.681179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.681983 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.681984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:04.682044 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.682049 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:57:04.682125 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:57:04.682183 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":7,"thread_start_us":132,"threads_started":1}
I20260929 18:57:04.682659 16997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e58b69c8484d56962531baa20a67e9" candidate_uuid: "d92a548a2eb44f1c95dea364a61da793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" is_pre_election: true
I20260929 18:57:04.682823 16997 raft_consensus.cc:2454] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92a548a2eb44f1c95dea364a61da793 in term 0.
I20260929 18:57:04.683064 16797 leader_election.cc:302] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9af8b3fa1540d98b16ebab8fea690e, d92a548a2eb44f1c95dea364a61da793; no voters:
I20260929 18:57:04.683032 17131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e58b69c8484d56962531baa20a67e9" candidate_uuid: "d92a548a2eb44f1c95dea364a61da793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d34f06210c444357ab5e4c2603ca26ff" is_pre_election: true
I20260929 18:57:04.683223 17131 raft_consensus.cc:2454] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92a548a2eb44f1c95dea364a61da793 in term 0.
I20260929 18:57:04.683225 17202 raft_consensus.cc:2781] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:57:04.683279 17202 raft_consensus.cc:482] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:57:04.683317 17202 raft_consensus.cc:3037] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:04.684136 17202 raft_consensus.cc:504] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.684383 17202 leader_election.cc:288] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [CANDIDATE]: Term 1 election: Requested vote from peers 7e9af8b3fa1540d98b16ebab8fea690e (127.29.71.130:35121), d34f06210c444357ab5e4c2603ca26ff (127.29.71.131:40833)
I20260929 18:57:04.684449 16997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e58b69c8484d56962531baa20a67e9" candidate_uuid: "d92a548a2eb44f1c95dea364a61da793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e9af8b3fa1540d98b16ebab8fea690e"
I20260929 18:57:04.684544 16997 raft_consensus.cc:3037] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:04.684556 17131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e58b69c8484d56962531baa20a67e9" candidate_uuid: "d92a548a2eb44f1c95dea364a61da793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d34f06210c444357ab5e4c2603ca26ff"
I20260929 18:57:04.684631 17131 raft_consensus.cc:3037] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:04.685366 17131 raft_consensus.cc:2454] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92a548a2eb44f1c95dea364a61da793 in term 1.
I20260929 18:57:04.685375 16997 raft_consensus.cc:2454] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92a548a2eb44f1c95dea364a61da793 in term 1.
I20260929 18:57:04.685555 16797 leader_election.cc:302] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e9af8b3fa1540d98b16ebab8fea690e, d92a548a2eb44f1c95dea364a61da793; no voters:
I20260929 18:57:04.685655 17202 raft_consensus.cc:2781] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:04.685822 17202 raft_consensus.cc:686] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [term 1 LEADER]: Becoming Leader. State: Replica: d92a548a2eb44f1c95dea364a61da793, State: Running, Role: LEADER
I20260929 18:57:04.685922 17202 consensus_queue.cc:237] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } }
I20260929 18:57:04.686811 16717 catalog_manager.cc:5495] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 reported cstate change: term changed from 0 to 1, leader changed from <none> to d92a548a2eb44f1c95dea364a61da793 (127.29.71.129). New cstate: current_term: 1 leader_uuid: "d92a548a2eb44f1c95dea364a61da793" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92a548a2eb44f1c95dea364a61da793" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 36811 } health_report { overall_health: HEALTHY } } }
I20260929 18:57:04.692678 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.690393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.29.71.129:36811)
0929 18:57:04.690437 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.690439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.690450 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:04.690484 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.690485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.690506 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.690577 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:04.690676 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.691165 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.691933 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.692131 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.692195 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.692380 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:04.692467 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:04.692558 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.692582 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.692600 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:57:04.694280 17131 raft_consensus.cc:1261] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 1 FOLLOWER]: Refusing update from remote peer d92a548a2eb44f1c95dea364a61da793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:04.694314 16997 raft_consensus.cc:1261] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 1 FOLLOWER]: Refusing update from remote peer d92a548a2eb44f1c95dea364a61da793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:04.694576 17202 consensus_queue.cc:1035] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d34f06210c444357ab5e4c2603ca26ff" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 40833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:04.694730 17204 consensus_queue.cc:1035] T f1e58b69c8484d56962531baa20a67e9 P d92a548a2eb44f1c95dea364a61da793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e9af8b3fa1540d98b16ebab8fea690e" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:04.697646 17207 mvcc.cc:205] Tried to move back new op lower bound from 7334740888345165824 to 7334740888314261504. Current Snapshot: MvccSnapshot[applied={T|T < 7334740888345165824 or (T in {7334740888345165824})}]
I20260929 18:57:04.698189 17210 mvcc.cc:205] Tried to move back new op lower bound from 7334740888345165824 to 7334740888314261504. Current Snapshot: MvccSnapshot[applied={T|T < 7334740888345165824}]
I20260929 18:57:04.699625 16717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:04.699728 16717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:04.700498 16717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19c24d8b9dcf4370ab077f48cc2afe6d: Sending DeleteTablet for 3 replicas of tablet f1e58b69c8484d56962531baa20a67e9
I20260929 18:57:04.700820 16977 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e58b69c8484d56962531baa20a67e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:04 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:42830
I20260929 18:57:04.701025 17111 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e58b69c8484d56962531baa20a67e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:04 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:53634
I20260929 18:57:04.701090 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 16780
I20260929 18:57:04.701220 16843 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e58b69c8484d56962531baa20a67e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:04 UTC) from {username='kudu', principal='kudu/127.29.71.190@KRBTEST.COM'} at 127.0.0.1:33562
I20260929 18:57:04.701666 17216 tablet_replica.cc:330] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: stopping tablet replica
I20260929 18:57:04.701784 17216 raft_consensus.cc:2229] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:57:04.701850 17217 tablet_replica.cc:330] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: stopping tablet replica
I20260929 18:57:04.701896 17216 raft_consensus.cc:2258] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:04.701939 17217 raft_consensus.cc:2229] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:57:04.702032 17217 raft_consensus.cc:2258] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:04.702481 17216 ts_tablet_manager.cc:1905] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:57:04.703547 17217 ts_tablet_manager.cc:1905] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:57:04.704425 17216 ts_tablet_manager.cc:1918] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:57:04.704501 17216 log.cc:1196] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/f1e58b69c8484d56962531baa20a67e9
I20260929 18:57:04.704771 17216 ts_tablet_manager.cc:1939] T f1e58b69c8484d56962531baa20a67e9 P 7e9af8b3fa1540d98b16ebab8fea690e: Deleting consensus metadata
I20260929 18:57:04.705161 17217 ts_tablet_manager.cc:1918] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:57:04.705184 16703 catalog_manager.cc:4841] TS 7e9af8b3fa1540d98b16ebab8fea690e (127.29.71.130:35121): tablet f1e58b69c8484d56962531baa20a67e9 (table test-table [id=349e58ee55104c428d65228a74f63e16]) successfully deleted
I20260929 18:57:04.705230 17217 log.cc:1196] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/f1e58b69c8484d56962531baa20a67e9
I20260929 18:57:04.705478 17217 ts_tablet_manager.cc:1939] T f1e58b69c8484d56962531baa20a67e9 P d34f06210c444357ab5e4c2603ca26ff: Deleting consensus metadata
I20260929 18:57:04.706022 16703 catalog_manager.cc:4841] TS d34f06210c444357ab5e4c2603ca26ff (127.29.71.131:40833): tablet f1e58b69c8484d56962531baa20a67e9 (table test-table [id=349e58ee55104c428d65228a74f63e16]) successfully deleted
W20260929 18:57:04.707113 16703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:36811 (error 108)
I20260929 18:57:04.707381 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 16914
I20260929 18:57:04.707572 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.707463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:36811 (local address 127.0.0.1:33592)
0929 18:57:04.707509 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:04.707538 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:36811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:57:04.707669 16703 catalog_manager.cc:4584] TS d92a548a2eb44f1c95dea364a61da793 (127.29.71.129:36811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1e58b69c8484d56962531baa20a67e9: Network error: Client connection negotiation failed: client connection to 127.29.71.129:36811: connect: Connection refused (error 111)
I20260929 18:57:04.712399 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17048
I20260929 18:57:04.717322 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 16686
2026-09-29T18:57:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T18:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:04Z Disabled control of system clock
I20260929 18:57:04.744616 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35171
--webserver_interface=127.29.71.190
--webserver_port=0
--builtin_ntp_servers=127.29.71.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:35171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:57:04.823412 17222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:04.823629 17222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:04.823650 17222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:04.824740 17222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:04.824774 17222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:04.824786 17222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:04.824798 17222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:04.824812 17222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:04.826066 17222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:35171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:35171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:04.826282 17222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:04.826534 17222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:04.831414 17230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.831414 17227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.831440 17228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:04.831800 17222 server_base.cc:1034] running on GCE node
I20260929 18:57:04.832115 17222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:04.832442 17222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:04.833630 17222 hybrid_clock.cc:648] HybridClock initialized: now 1790708224833606 us; error 43 us; skew 500 ppm
I20260929 18:57:04.835219 17222 webserver.cc:460] Webserver started at http://127.29.71.190:35011/ using document root <none> and password file <none>
I20260929 18:57:04.835515 17222 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:04.835595 17222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:04.835767 17222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:04.837062 17222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "cb9defb891f74dfab71005f21f1c6ab2"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "f5d22cdfd0d8b090ef2d77d2578e219c"
server_key_iv: "80147766c033a58689cf8f50494b6f81"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.837459 17222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "cb9defb891f74dfab71005f21f1c6ab2"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "f5d22cdfd0d8b090ef2d77d2578e219c"
server_key_iv: "80147766c033a58689cf8f50494b6f81"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.839136 17222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260929 18:57:04.840022 17236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:04.840281 17222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:04.840387 17222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "cb9defb891f74dfab71005f21f1c6ab2"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "f5d22cdfd0d8b090ef2d77d2578e219c"
server_key_iv: "80147766c033a58689cf8f50494b6f81"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.840479 17222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:04.861056 17222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:04.861429 17222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:04.861600 17222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:04.865816 17222 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:35171
I20260929 18:57:04.865886 17288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:35171 every 8 connection(s)
I20260929 18:57:04.866282 17222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:04.867437 17289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:04.868860 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17222
I20260929 18:57:04.868984 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:04.869210 29982 external_mini_cluster.cc:1428] Setting key dff806f5faf29abac5075df87da40bb6
I20260929 18:57:04.870393 17289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2: Bootstrap starting.
I20260929 18:57:04.871011 17289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:04.871418 17289 log.cc:824] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:04.872236 17289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2: No bootstrap required, opened a new log
I20260929 18:57:04.874102 17289 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9defb891f74dfab71005f21f1c6ab2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35171 } }
I20260929 18:57:04.874348 17289 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:04.874408 17289 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9defb891f74dfab71005f21f1c6ab2, State: Initialized, Role: FOLLOWER
I20260929 18:57:04.874542 17289 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9defb891f74dfab71005f21f1c6ab2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35171 } }
I20260929 18:57:04.874650 17289 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:04.874719 17289 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:04.874766 17289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:04.875416 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:04.869834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.29.71.190:35171)
0929 18:57:04.870059 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:04.870070 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:04.870107 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:57:04.870680 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:04.870687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:04.870766 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:04.870927 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:04.871220 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.872388 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.873264 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:04.873723 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:04.873894 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:04.874682 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:04.874687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:04.874687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:04.874936 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:04.874973 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:04.874985 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:04.875068 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:04.875153 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 18:57:04.875237 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260929 18:57:04.875393 17289 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9defb891f74dfab71005f21f1c6ab2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35171 } }
I20260929 18:57:04.875563 17289 leader_election.cc:302] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb9defb891f74dfab71005f21f1c6ab2; no voters:
I20260929 18:57:04.875746 17289 leader_election.cc:288] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:04.875779 17294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:04.875898 17294 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [term 1 LEADER]: Becoming Leader. State: Replica: cb9defb891f74dfab71005f21f1c6ab2, State: Running, Role: LEADER
I20260929 18:57:04.875973 17294 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9defb891f74dfab71005f21f1c6ab2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35171 } }
I20260929 18:57:04.876016 17289 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:04.876510 17296 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb9defb891f74dfab71005f21f1c6ab2. Latest consensus state: current_term: 1 leader_uuid: "cb9defb891f74dfab71005f21f1c6ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9defb891f74dfab71005f21f1c6ab2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35171 } } }
I20260929 18:57:04.876502 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb9defb891f74dfab71005f21f1c6ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9defb891f74dfab71005f21f1c6ab2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 35171 } } }
I20260929 18:57:04.876613 17296 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:04.876619 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:04.876850 17301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:04.877400 17301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:04.879236 17301 catalog_manager.cc:1347] Generated new cluster ID: 91d49221f2a4425284a4efee53da7bc5
I20260929 18:57:04.879299 17301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:04.883394 17301 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:04.883901 17301 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:04.891490 17301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2: Generated new TSK 0
I20260929 18:57:04.891681 17301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:04.899552 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.129:0
--local_ip_for_outbound_sockets=127.29.71.129
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35171
--builtin_ntp_servers=127.29.71.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:57:04.983218 17313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:04.983438 17313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:04.983460 17313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:04.984687 17313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:04.984735 17313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:04.984763 17313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.129
I20260929 18:57:04.986119 17313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.71.129
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.129
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:04.986450 17313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:04.986795 17313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:04.992606 17319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.992602 17321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:04.992602 17318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:04.992889 17313 server_base.cc:1034] running on GCE node
I20260929 18:57:04.993104 17313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:04.993343 17313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:04.994505 17313 hybrid_clock.cc:648] HybridClock initialized: now 1790708224994485 us; error 33 us; skew 500 ppm
I20260929 18:57:04.995810 17313 webserver.cc:460] Webserver started at http://127.29.71.129:42353/ using document root <none> and password file <none>
I20260929 18:57:04.996066 17313 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:04.996137 17313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:04.996284 17313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:04.997365 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data/instance:
uuid: "04c6afb4f2bc401e9f1686057d5d5fa0"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "6f24df55b2eb19bf6e9acd7036a78c07"
server_key_iv: "6023b6fb3552e53b40ae192feb5421d3"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.997730 17313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance:
uuid: "04c6afb4f2bc401e9f1686057d5d5fa0"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "6f24df55b2eb19bf6e9acd7036a78c07"
server_key_iv: "6023b6fb3552e53b40ae192feb5421d3"
server_key_version: "encryptionkey@0"
I20260929 18:57:04.999195 17313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:57:05.000095 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.000330 17313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:05.000411 17313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/wal
uuid: "04c6afb4f2bc401e9f1686057d5d5fa0"
format_stamp: "Formatted at 2026-09-29 18:57:04 on dist-test-slave-q2kh"
server_key: "6f24df55b2eb19bf6e9acd7036a78c07"
server_key_iv: "6023b6fb3552e53b40ae192feb5421d3"
server_key_version: "encryptionkey@0"
I20260929 18:57:05.000478 17313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:05.020578 17313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:05.020870 17313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:05.020975 17313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:05.021175 17313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:05.021546 17313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:05.021580 17313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.021634 17313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:05.021659 17313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.027397 17313 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.129:37775
I20260929 18:57:05.027487 17440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.129:37775 every 8 connection(s)
I20260929 18:57:05.027824 17313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/data/info.pb
I20260929 18:57:05.033586 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.028617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.129:35119 (local address 127.29.71.190:35171)
0929 18:57:05.028776 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.028779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.029075 (+ 296us) server_negotiation.cc:407] Connection header received
0929 18:57:05.029634 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.029635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.029648 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.029706 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:05.030461 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.030976 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.032101 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.032312 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.032406 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.033276 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:05.033277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:05.033277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:05.033282 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:05.033298 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:05.033302 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:05.033378 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.033476 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.033503 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:57:05.033821 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.028618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35171 (local address 127.29.71.129:35119)
0929 18:57:05.028903 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.028951 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.029518 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.029783 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.029788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.029805 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:05.030359 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.030365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.031077 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.031078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.032006 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.032009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.032417 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.032418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.032549 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.033124 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:57:05.033125 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:05.033154 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:57:05.033177 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:05.033379 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:05.033386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:05.033389 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.033479 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.033579 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
I20260929 18:57:05.034410 17441 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35171
I20260929 18:57:05.034514 17441 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:05.034708 17441 heartbeater.cc:511] Master 127.29.71.190:35171 requested a full tablet report, sending...
I20260929 18:57:05.034711 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17313
I20260929 18:57:05.034853 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-0/wal/instance
I20260929 18:57:05.035101 29982 external_mini_cluster.cc:1428] Setting key 450ef57f98c1339544b0e75a1c8da62d
I20260929 18:57:05.035300 17253 ts_manager.cc:194] Registered new tserver with Master: 04c6afb4f2bc401e9f1686057d5d5fa0 (127.29.71.129:37775)
I20260929 18:57:05.036188 17253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.129:35119
I20260929 18:57:05.036208 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.130:0
--local_ip_for_outbound_sockets=127.29.71.130
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35171
--builtin_ntp_servers=127.29.71.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:57:05.117792 17444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:05.118055 17444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:05.118091 17444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:05.119328 17444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:05.119395 17444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:05.119436 17444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.130
I20260929 18:57:05.120796 17444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.71.130
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.130
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:05.121073 17444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:05.121467 17444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:05.126962 17449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:05.126955 17450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:05.126955 17452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:05.127033 17444 server_base.cc:1034] running on GCE node
I20260929 18:57:05.127481 17444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:05.127753 17444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:05.128899 17444 hybrid_clock.cc:648] HybridClock initialized: now 1790708225128848 us; error 68 us; skew 500 ppm
I20260929 18:57:05.130355 17444 webserver.cc:460] Webserver started at http://127.29.71.130:35271/ using document root <none> and password file <none>
I20260929 18:57:05.130622 17444 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:05.130672 17444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:05.130796 17444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:05.132037 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data/instance:
uuid: "bc3a98de21f8450b88392a172c9e5ccd"
format_stamp: "Formatted at 2026-09-29 18:57:05 on dist-test-slave-q2kh"
server_key: "4292a3c01cee8daf0fea374bed33253a"
server_key_iv: "86672d37a29639e5b9e9c30a25e34b0f"
server_key_version: "encryptionkey@0"
I20260929 18:57:05.132426 17444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance:
uuid: "bc3a98de21f8450b88392a172c9e5ccd"
format_stamp: "Formatted at 2026-09-29 18:57:05 on dist-test-slave-q2kh"
server_key: "4292a3c01cee8daf0fea374bed33253a"
server_key_iv: "86672d37a29639e5b9e9c30a25e34b0f"
server_key_version: "encryptionkey@0"
I20260929 18:57:05.133759 17444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:05.134598 17458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.134796 17444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 18:57:05.134868 17444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal
uuid: "bc3a98de21f8450b88392a172c9e5ccd"
format_stamp: "Formatted at 2026-09-29 18:57:05 on dist-test-slave-q2kh"
server_key: "4292a3c01cee8daf0fea374bed33253a"
server_key_iv: "86672d37a29639e5b9e9c30a25e34b0f"
server_key_version: "encryptionkey@0"
I20260929 18:57:05.134979 17444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:05.150835 17444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:05.151247 17444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:05.151403 17444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:05.151654 17444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:05.152091 17444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:05.152165 17444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.152201 17444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:05.152231 17444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.159322 17444 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.130:39489
I20260929 18:57:05.159379 17571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.130:39489 every 8 connection(s)
I20260929 18:57:05.159700 17444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/data/info.pb
I20260929 18:57:05.161368 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17444
I20260929 18:57:05.161494 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal/instance
I20260929 18:57:05.161697 29982 external_mini_cluster.cc:1428] Setting key 68b889ea36c4a78525c01d61c7190f10
I20260929 18:57:05.162806 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.71.131:0
--local_ip_for_outbound_sockets=127.29.71.131
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35171
--builtin_ntp_servers=127.29.71.148:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:57:05.165732 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.160455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.130:37665 (local address 127.29.71.190:35171)
0929 18:57:05.160539 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.160541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.160746 (+ 205us) server_negotiation.cc:407] Connection header received
0929 18:57:05.161304 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.161305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.161318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.161373 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:05.162309 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.163135 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.164271 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.164452 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.164507 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.165485 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:05.165487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:05.165487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:05.165492 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:05.165505 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:05.165509 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:05.165553 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.165630 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.165656 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:57:05.165982 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.160424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35171 (local address 127.29.71.130:37665)
0929 18:57:05.160627 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.160662 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.161172 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.161453 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.161457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.161484 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:05.162223 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.162230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.163224 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.163225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.164179 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.164182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.164535 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.164536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.164700 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.165290 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:57:05.165290 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:05.165319 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:57:05.165362 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:05.165564 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:05.165569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:05.165572 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.165636 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.165752 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260929 18:57:05.166492 17572 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35171
I20260929 18:57:05.166584 17572 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:05.166769 17572 heartbeater.cc:511] Master 127.29.71.190:35171 requested a full tablet report, sending...
I20260929 18:57:05.167133 17253 ts_manager.cc:194] Registered new tserver with Master: bc3a98de21f8450b88392a172c9e5ccd (127.29.71.130:39489)
I20260929 18:57:05.167518 17253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.130:37665
W20260929 18:57:05.243997 17575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:05.244216 17575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:05.244248 17575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:05.245420 17575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:05.245481 17575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:05.245520 17575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.71.131
I20260929 18:57:05.246876 17575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.71.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.71.131
--webserver_port=0
--tserver_master_addrs=127.29.71.190:35171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.71.131
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:05.247123 17575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:05.247397 17575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:05.252478 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:05.252461 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:05.252440 17583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:05.252650 17575 server_base.cc:1034] running on GCE node
I20260929 18:57:05.253016 17575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:05.253244 17575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:05.254438 17575 hybrid_clock.cc:648] HybridClock initialized: now 1790708225254426 us; error 31 us; skew 500 ppm
I20260929 18:57:05.255710 17575 webserver.cc:460] Webserver started at http://127.29.71.131:36923/ using document root <none> and password file <none>
I20260929 18:57:05.255970 17575 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:05.256039 17575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:05.256187 17575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:05.257293 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data/instance:
uuid: "a4eefc08f7ee4556b552bca6c408bb34"
format_stamp: "Formatted at 2026-09-29 18:57:05 on dist-test-slave-q2kh"
server_key: "e9023bb3a8b4f5f2248fb27016231262"
server_key_iv: "650fc880971477c4b08e103034c05b63"
server_key_version: "encryptionkey@0"
I20260929 18:57:05.257651 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance:
uuid: "a4eefc08f7ee4556b552bca6c408bb34"
format_stamp: "Formatted at 2026-09-29 18:57:05 on dist-test-slave-q2kh"
server_key: "e9023bb3a8b4f5f2248fb27016231262"
server_key_iv: "650fc880971477c4b08e103034c05b63"
server_key_version: "encryptionkey@0"
I20260929 18:57:05.259006 17575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:05.259794 17589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.259979 17575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:05.260078 17575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal
uuid: "a4eefc08f7ee4556b552bca6c408bb34"
format_stamp: "Formatted at 2026-09-29 18:57:05 on dist-test-slave-q2kh"
server_key: "e9023bb3a8b4f5f2248fb27016231262"
server_key_iv: "650fc880971477c4b08e103034c05b63"
server_key_version: "encryptionkey@0"
I20260929 18:57:05.260156 17575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:05.274085 17575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:05.274447 17575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:05.274608 17575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:05.274899 17575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:57:05.275233 17575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:57:05.275286 17575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.275324 17575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:57:05.275367 17575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:05.281474 17575 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.131:45423
I20260929 18:57:05.281533 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.131:45423 every 8 connection(s)
I20260929 18:57:05.281898 17575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/data/info.pb
I20260929 18:57:05.287648 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.282697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:40711 (local address 127.29.71.190:35171)
0929 18:57:05.282746 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.282748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.283055 (+ 307us) server_negotiation.cc:407] Connection header received
0929 18:57:05.283514 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.283515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.283526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.283591 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:05.284277 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.285184 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.286290 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.286484 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.286546 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.287364 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:05.287366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:05.287366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:05.287370 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:05.287386 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:05.287390 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:05.287439 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.287567 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.287590 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:05.287902 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.282702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:35171 (local address 127.29.71.131:40711)
0929 18:57:05.282899 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.282933 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.283427 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.283658 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.283661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.283669 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:05.284176 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.284180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.285337 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.285338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.286186 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.286188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.286563 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.286564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.286672 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.287196 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:57:05.287197 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:05.287245 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:57:05.287269 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:05.287478 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:05.287484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:05.287487 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.287563 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.287652 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260929 18:57:05.288450 17703 heartbeater.cc:347] Connected to a master server at 127.29.71.190:35171
I20260929 18:57:05.288524 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17575
I20260929 18:57:05.288565 17703 heartbeater.cc:464] Registering TS with master...
I20260929 18:57:05.288635 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal/instance
I20260929 18:57:05.288781 17703 heartbeater.cc:511] Master 127.29.71.190:35171 requested a full tablet report, sending...
I20260929 18:57:05.288913 29982 external_mini_cluster.cc:1428] Setting key c3281199829edfd80ea5985a3c093848
I20260929 18:57:05.289194 17253 ts_manager.cc:194] Registered new tserver with Master: a4eefc08f7ee4556b552bca6c408bb34 (127.29.71.131:45423)
I20260929 18:57:05.289758 17253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.71.131:40711
I20260929 18:57:05.289861 29982 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:57:05.294385 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.291522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.29.71.190:35171)
0929 18:57:05.291563 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.291565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.291755 (+ 190us) server_negotiation.cc:407] Connection header received
0929 18:57:05.291776 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.291777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.291787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.291841 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:05.291994 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.292443 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.293209 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.293386 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.293453 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.294050 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:05.294051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:05.294052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:05.294056 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:05.294070 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:05.294074 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:05.294122 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.294252 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.294282 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:05.295830 17253 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:57:05.297477 17253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:57:05.304520 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.300127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.29.71.131:45423)
0929 18:57:05.300379 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.300384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.300395 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:05.300445 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.300447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.300465 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.300765 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:05.300820 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.301960 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.303409 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.304066 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.304171 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.304275 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.304383 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.304424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":134,"threads_started":1}
I20260929 18:57:05.304562 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.299807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:39489 (local address 127.0.0.1:45970)
0929 18:57:05.299999 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.300056 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.300143 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.300745 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.300746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.300748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:05.300800 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.300802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.301769 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.301770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.303267 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.303269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.304295 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.304297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.304360 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.304363 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.304421 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.304476 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":61,"thread_start_us":112,"threads_started":1}
I20260929 18:57:05.304656 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.300363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.29.71.129:37775)
0929 18:57:05.300541 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.300546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.300557 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:05.300597 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.300599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.300614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.300860 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:05.300926 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.301995 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.303245 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.304061 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.304146 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.304254 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.304521 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.304563 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260929 18:57:05.304827 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.300125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.29.71.130:39489)
0929 18:57:05.300585 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.300590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.300601 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:05.300643 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.300645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.300664 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.300748 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:05.300862 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.301659 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.303383 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.304106 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.304171 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.304255 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.304682 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.304731 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":85,"threads_started":1}
I20260929 18:57:05.305163 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.299807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.131:45423 (local address 127.0.0.1:42810)
0929 18:57:05.300005 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.300055 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.300142 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.300598 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.300601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.300604 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:05.300680 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.300683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.302059 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.302060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.303315 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.303317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.304174 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.304175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.304267 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.304281 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.304363 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.304420 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":130,"threads_started":1}
I20260929 18:57:05.305369 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.299812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37775 (local address 127.0.0.1:48328)
0929 18:57:05.300270 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.300280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.300322 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.300742 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.300743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.300745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:05.300777 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.300779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.302090 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.302091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.303128 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.303130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.304297 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.304298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.304358 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.304365 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.304421 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.304476 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260929 18:57:05.305610 17506 tablet_service.cc:1467] Processing CreateTablet for tablet d5b1e296318e4c6fb2bfd1729773d29a (DEFAULT_TABLE table=test-table [id=0fcf895843fe40db9a48307333eb544d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:05.306032 17506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b1e296318e4c6fb2bfd1729773d29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:05.305944 17637 tablet_service.cc:1467] Processing CreateTablet for tablet d5b1e296318e4c6fb2bfd1729773d29a (DEFAULT_TABLE table=test-table [id=0fcf895843fe40db9a48307333eb544d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:05.306306 17637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b1e296318e4c6fb2bfd1729773d29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:05.306669 17375 tablet_service.cc:1467] Processing CreateTablet for tablet d5b1e296318e4c6fb2bfd1729773d29a (DEFAULT_TABLE table=test-table [id=0fcf895843fe40db9a48307333eb544d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:57:05.306906 17375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b1e296318e4c6fb2bfd1729773d29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:05.309520 17717 tablet_bootstrap.cc:492] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0: Bootstrap starting.
I20260929 18:57:05.309930 17718 tablet_bootstrap.cc:492] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Bootstrap starting.
I20260929 18:57:05.310230 17717 tablet_bootstrap.cc:654] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:05.310348 17719 tablet_bootstrap.cc:492] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Bootstrap starting.
I20260929 18:57:05.310567 17717 log.cc:824] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:05.310673 17718 tablet_bootstrap.cc:654] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:05.311005 17718 log.cc:824] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:05.311053 17719 tablet_bootstrap.cc:654] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:05.311318 17717 tablet_bootstrap.cc:492] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0: No bootstrap required, opened a new log
I20260929 18:57:05.311370 17719 log.cc:824] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:05.311391 17717 ts_tablet_manager.cc:1397] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:57:05.311748 17718 tablet_bootstrap.cc:492] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: No bootstrap required, opened a new log
I20260929 18:57:05.311841 17718 ts_tablet_manager.cc:1397] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:57:05.312112 17719 tablet_bootstrap.cc:492] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: No bootstrap required, opened a new log
I20260929 18:57:05.312180 17719 ts_tablet_manager.cc:1397] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:57:05.313310 17717 raft_consensus.cc:348] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.313463 17717 raft_consensus.cc:374] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:05.313504 17717 raft_consensus.cc:729] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c6afb4f2bc401e9f1686057d5d5fa0, State: Initialized, Role: FOLLOWER
I20260929 18:57:05.313632 17717 consensus_queue.cc:260] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.313905 17717 ts_tablet_manager.cc:1428] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 18:57:05.314244 17441 heartbeater.cc:503] Master 127.29.71.190:35171 was elected leader, sending a full tablet report...
I20260929 18:57:05.314698 17718 raft_consensus.cc:348] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.314870 17718 raft_consensus.cc:374] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:05.314918 17718 raft_consensus.cc:729] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3a98de21f8450b88392a172c9e5ccd, State: Initialized, Role: FOLLOWER
I20260929 18:57:05.314966 17719 raft_consensus.cc:348] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.315068 17718 consensus_queue.cc:260] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.315120 17719 raft_consensus.cc:374] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:05.315158 17719 raft_consensus.cc:729] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4eefc08f7ee4556b552bca6c408bb34, State: Initialized, Role: FOLLOWER
I20260929 18:57:05.315289 17719 consensus_queue.cc:260] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.315336 17718 ts_tablet_manager.cc:1428] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260929 18:57:05.315371 17572 heartbeater.cc:503] Master 127.29.71.190:35171 was elected leader, sending a full tablet report...
I20260929 18:57:05.315582 17719 ts_tablet_manager.cc:1428] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 18:57:05.315600 17703 heartbeater.cc:503] Master 127.29.71.190:35171 was elected leader, sending a full tablet report...
W20260929 18:57:05.410329 17573 tablet.cc:2369] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:57:05.498226 17725 raft_consensus.cc:482] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:57:05.498360 17725 raft_consensus.cc:504] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.498804 17725 leader_election.cc:288] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc3a98de21f8450b88392a172c9e5ccd (127.29.71.130:39489), 04c6afb4f2bc401e9f1686057d5d5fa0 (127.29.71.129:37775)
I20260929 18:57:05.501892 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.499156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:39795 (local address 127.29.71.129:37775)
0929 18:57:05.499246 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.499248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.499327 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:57:05.499387 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.499388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.499408 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.499486 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:05.499713 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.500232 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.501161 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.501525 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.501612 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.501620 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.501778 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.501816 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:57:05.501916 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.499086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.71.131:44399 (local address 127.29.71.130:39489)
0929 18:57:05.499135 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.499137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.499149 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:57:05.499292 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.499293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.499301 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.499390 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:57:05.499579 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.500170 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.501160 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.501527 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.501623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.501632 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.501788 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.501827 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:57:05.501919 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.499012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.130:39489 (local address 127.29.71.131:44399)
0929 18:57:05.499092 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.499105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.499153 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.499394 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.499395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.499397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:05.499503 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.499505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.500272 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.500273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.501064 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.501066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.501623 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.501624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.501697 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.501700 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.501781 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.501828 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 18:57:05.501941 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.499054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37775 (local address 127.29.71.131:39795)
0929 18:57:05.499233 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.499242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:05.499338 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:05.499488 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:05.499489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:05.499491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:57:05.499576 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.499578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.500327 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.500328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.501068 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:05.501069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.501618 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.501619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:05.501698 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.501700 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:57:05.501760 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:57:05.501801 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260929 18:57:05.502316 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b1e296318e4c6fb2bfd1729773d29a" candidate_uuid: "a4eefc08f7ee4556b552bca6c408bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3a98de21f8450b88392a172c9e5ccd" is_pre_election: true
I20260929 18:57:05.502475 17516 raft_consensus.cc:2454] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4eefc08f7ee4556b552bca6c408bb34 in term 0.
I20260929 18:57:05.502480 17395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b1e296318e4c6fb2bfd1729773d29a" candidate_uuid: "a4eefc08f7ee4556b552bca6c408bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" is_pre_election: true
I20260929 18:57:05.502681 17395 raft_consensus.cc:2454] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4eefc08f7ee4556b552bca6c408bb34 in term 0.
I20260929 18:57:05.502712 17592 leader_election.cc:302] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4eefc08f7ee4556b552bca6c408bb34, bc3a98de21f8450b88392a172c9e5ccd; no voters:
I20260929 18:57:05.502856 17725 raft_consensus.cc:2781] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:57:05.502935 17725 raft_consensus.cc:482] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:57:05.502955 17725 raft_consensus.cc:3037] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:05.503618 17725 raft_consensus.cc:504] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.503791 17725 leader_election.cc:288] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [CANDIDATE]: Term 1 election: Requested vote from peers bc3a98de21f8450b88392a172c9e5ccd (127.29.71.130:39489), 04c6afb4f2bc401e9f1686057d5d5fa0 (127.29.71.129:37775)
I20260929 18:57:05.503901 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b1e296318e4c6fb2bfd1729773d29a" candidate_uuid: "a4eefc08f7ee4556b552bca6c408bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3a98de21f8450b88392a172c9e5ccd"
I20260929 18:57:05.503993 17395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b1e296318e4c6fb2bfd1729773d29a" candidate_uuid: "a4eefc08f7ee4556b552bca6c408bb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0"
I20260929 18:57:05.504031 17516 raft_consensus.cc:3037] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:05.504066 17395 raft_consensus.cc:3037] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:05.504985 17395 raft_consensus.cc:2454] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4eefc08f7ee4556b552bca6c408bb34 in term 1.
I20260929 18:57:05.504985 17516 raft_consensus.cc:2454] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4eefc08f7ee4556b552bca6c408bb34 in term 1.
I20260929 18:57:05.505169 17592 leader_election.cc:302] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4eefc08f7ee4556b552bca6c408bb34, bc3a98de21f8450b88392a172c9e5ccd; no voters:
I20260929 18:57:05.505285 17725 raft_consensus.cc:2781] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:05.505426 17725 raft_consensus.cc:686] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 1 LEADER]: Becoming Leader. State: Replica: a4eefc08f7ee4556b552bca6c408bb34, State: Running, Role: LEADER
I20260929 18:57:05.505530 17725 consensus_queue.cc:237] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } }
I20260929 18:57:05.506430 17253 catalog_manager.cc:5495] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4eefc08f7ee4556b552bca6c408bb34 (127.29.71.131). New cstate: current_term: 1 leader_uuid: "a4eefc08f7ee4556b552bca6c408bb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4eefc08f7ee4556b552bca6c408bb34" member_type: VOTER last_known_addr { host: "127.29.71.131" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 } health_report { overall_health: UNKNOWN } } }
W20260929 18:57:05.528571 17442 tablet.cc:2369] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:57:05.530529 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.528104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.29.71.131:45423)
0929 18:57:05.528153 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:05.528155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:05.528162 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:57:05.528199 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:05.528200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:05.528218 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:05.528284 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:05.528466 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.529068 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.529879 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:05.530061 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:05.530117 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:05.530251 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:05.530352 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:05.530412 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:05.530443 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 18:57:05.530457 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:57:05.532413 17395 raft_consensus.cc:1261] T d5b1e296318e4c6fb2bfd1729773d29a P 04c6afb4f2bc401e9f1686057d5d5fa0 [term 1 FOLLOWER]: Refusing update from remote peer a4eefc08f7ee4556b552bca6c408bb34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 18:57:05.532429 17704 tablet.cc:2369] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:57:05.532475 17516 raft_consensus.cc:1261] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 1 FOLLOWER]: Refusing update from remote peer a4eefc08f7ee4556b552bca6c408bb34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:05.532783 17727 consensus_queue.cc:1035] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc3a98de21f8450b88392a172c9e5ccd" member_type: VOTER last_known_addr { host: "127.29.71.130" port: 39489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:05.532933 17725 consensus_queue.cc:1035] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c6afb4f2bc401e9f1686057d5d5fa0" member_type: VOTER last_known_addr { host: "127.29.71.129" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:05.535611 17731 mvcc.cc:205] Tried to move back new op lower bound from 7334740891777740800 to 7334740891671367680. Current Snapshot: MvccSnapshot[applied={T|T < 7334740891777740800 or (T in {7334740891777740800})}]
I20260929 18:57:05.537549 17253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:05.537659 17253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:57:05.538448 17253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb9defb891f74dfab71005f21f1c6ab2: Sending DeleteTablet for 3 replicas of tablet d5b1e296318e4c6fb2bfd1729773d29a
I20260929 18:57:05.538779 17506 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b1e296318e4c6fb2bfd1729773d29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:05 UTC) from {username='slave'} at 127.0.0.1:45970
I20260929 18:57:05.538864 17637 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b1e296318e4c6fb2bfd1729773d29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:05 UTC) from {username='slave'} at 127.0.0.1:42810
I20260929 18:57:05.538884 17375 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b1e296318e4c6fb2bfd1729773d29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:57:05 UTC) from {username='slave'} at 127.0.0.1:48328
I20260929 18:57:05.539049 17739 tablet_replica.cc:330] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: stopping tablet replica
I20260929 18:57:05.539145 17739 raft_consensus.cc:2229] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:57:05.539243 17739 raft_consensus.cc:2258] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:05.539426 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17313
I20260929 18:57:05.540202 17739 ts_tablet_manager.cc:1905] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:57:05.541967 17739 ts_tablet_manager.cc:1918] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:57:05.542071 17739 log.cc:1196] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-1/wal/wals/d5b1e296318e4c6fb2bfd1729773d29a
I20260929 18:57:05.542464 17739 ts_tablet_manager.cc:1939] T d5b1e296318e4c6fb2bfd1729773d29a P bc3a98de21f8450b88392a172c9e5ccd: Deleting consensus metadata
I20260929 18:57:05.543404 17239 catalog_manager.cc:4841] TS bc3a98de21f8450b88392a172c9e5ccd (127.29.71.130:39489): tablet d5b1e296318e4c6fb2bfd1729773d29a (table test-table [id=0fcf895843fe40db9a48307333eb544d]) successfully deleted
I20260929 18:57:05.543422 17740 tablet_replica.cc:330] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: stopping tablet replica
I20260929 18:57:05.543529 17740 raft_consensus.cc:2229] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:57:05.543630 17740 raft_consensus.cc:2258] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:57:05.544164 17740 ts_tablet_manager.cc:1905] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:57:05.545811 17740 ts_tablet_manager.cc:1918] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:57:05.545897 17740 log.cc:1196] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Deleting WAL directory at /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790707954762284-29982-0/minicluster-data/ts-2/wal/wals/d5b1e296318e4c6fb2bfd1729773d29a
I20260929 18:57:05.546116 17740 ts_tablet_manager.cc:1939] T d5b1e296318e4c6fb2bfd1729773d29a P a4eefc08f7ee4556b552bca6c408bb34: Deleting consensus metadata
I20260929 18:57:05.546540 17240 catalog_manager.cc:4841] TS a4eefc08f7ee4556b552bca6c408bb34 (127.29.71.131:45423): tablet d5b1e296318e4c6fb2bfd1729773d29a (table test-table [id=0fcf895843fe40db9a48307333eb544d]) successfully deleted
W20260929 18:57:05.547725 17237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.71.129:37775 (error 108)
I20260929 18:57:05.547945 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17444
I20260929 18:57:05.548553 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:05.548052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.129:37775 (local address 127.0.0.1:48338)
0929 18:57:05.548473 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:05.548512 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400}
W20260929 18:57:05.548692 17237 catalog_manager.cc:4584] TS 04c6afb4f2bc401e9f1686057d5d5fa0 (127.29.71.129:37775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5b1e296318e4c6fb2bfd1729773d29a: Network error: Client connection negotiation failed: client connection to 127.29.71.129:37775: connect: Connection refused (error 111)
I20260929 18:57:05.553634 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17575
I20260929 18:57:05.558780 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17222
2026-09-29T18:57:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (840 ms)
[----------] 6 tests from EncryptionPolicyTest (8858 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:05 dist-test-slave-q2kh krb5kdc[17745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:05 dist-test-slave-q2kh krb5kdc[17745](info): set up 2 sockets
Sep 29 18:57:05 dist-test-slave-q2kh krb5kdc[17745](info): commencing operation
krb5kdc: starting...
W20260929 18:57:06.602424 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:06 dist-test-slave-q2kh krb5kdc[17745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:06.721294 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:06.803635 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:06.803853 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:06.803877 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:06.805066 17758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:06.805112 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:06.805127 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:06.805164 17758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:06.805178 17758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:06.806802 17758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40209
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:06.807070 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:06.807330 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:06.812162 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:06.812170 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:06.812170 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:06.812633 17758 server_base.cc:1034] running on GCE node
I20260929 18:57:06.812949 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:06.813275 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:06.814447 17758 hybrid_clock.cc:648] HybridClock initialized: now 1790708226814445 us; error 42 us; skew 500 ppm
Sep 29 18:57:06 dist-test-slave-q2kh krb5kdc[17745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:06.816998 17758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:06.817494 17758 webserver.cc:460] Webserver started at http://127.0.0.1:43085/ using document root <none> and password file <none>
I20260929 18:57:06.817754 17758 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:06.817802 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:06.817931 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:06.819144 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "e41a89e4fc074253911b2690834e7c03"
format_stamp: "Formatted at 2026-09-29 18:57:06 on dist-test-slave-q2kh"
server_key: "9ef10585d13f58ee1c3a0564ef743dc3"
server_key_iv: "5afaa29b9d4782184190ab645c593b5f"
server_key_version: "encryptionkey@0"
I20260929 18:57:06.819516 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "e41a89e4fc074253911b2690834e7c03"
format_stamp: "Formatted at 2026-09-29 18:57:06 on dist-test-slave-q2kh"
server_key: "9ef10585d13f58ee1c3a0564ef743dc3"
server_key_iv: "5afaa29b9d4782184190ab645c593b5f"
server_key_version: "encryptionkey@0"
I20260929 18:57:06.820871 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:57:06.821655 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:06.821923 17758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:06.822029 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "e41a89e4fc074253911b2690834e7c03"
format_stamp: "Formatted at 2026-09-29 18:57:06 on dist-test-slave-q2kh"
server_key: "9ef10585d13f58ee1c3a0564ef743dc3"
server_key_iv: "5afaa29b9d4782184190ab645c593b5f"
server_key_version: "encryptionkey@0"
I20260929 18:57:06.822105 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:06.845928 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:06.848193 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:06.848389 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:06.852725 17758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40209
I20260929 18:57:06.852866 17825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40209 every 8 connection(s)
I20260929 18:57:06.853180 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:06.854035 17826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:06.856330 17826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03: Bootstrap starting.
I20260929 18:57:06.856650 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17758
I20260929 18:57:06.856814 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:06.857010 29982 external_mini_cluster.cc:1428] Setting key b4db2faffb1572c436102f4ec55e17e9
I20260929 18:57:06.857122 17826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:06.857445 17826 log.cc:824] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:06.858276 17826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03: No bootstrap required, opened a new log
I20260929 18:57:06.860110 17826 raft_consensus.cc:348] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41a89e4fc074253911b2690834e7c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40209 } }
I20260929 18:57:06.860272 17826 raft_consensus.cc:374] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:06.860319 17826 raft_consensus.cc:729] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41a89e4fc074253911b2690834e7c03, State: Initialized, Role: FOLLOWER
I20260929 18:57:06.860484 17826 consensus_queue.cc:260] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41a89e4fc074253911b2690834e7c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40209 } }
I20260929 18:57:06.860567 17826 raft_consensus.cc:388] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:06.860610 17826 raft_consensus.cc:482] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:06.860656 17826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:06.861207 17826 raft_consensus.cc:504] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41a89e4fc074253911b2690834e7c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40209 } }
I20260929 18:57:06.861325 17826 leader_election.cc:302] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e41a89e4fc074253911b2690834e7c03; no voters:
I20260929 18:57:06.861545 17826 leader_election.cc:288] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:06.861567 17831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:06.861677 17831 raft_consensus.cc:686] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [term 1 LEADER]: Becoming Leader. State: Replica: e41a89e4fc074253911b2690834e7c03, State: Running, Role: LEADER
I20260929 18:57:06.861759 17831 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41a89e4fc074253911b2690834e7c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40209 } }
I20260929 18:57:06.861821 17826 sys_catalog.cc:564] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:06.862103 17832 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41a89e4fc074253911b2690834e7c03. Latest consensus state: current_term: 1 leader_uuid: "e41a89e4fc074253911b2690834e7c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41a89e4fc074253911b2690834e7c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40209 } } }
I20260929 18:57:06.862263 17832 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:06.862102 17833 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e41a89e4fc074253911b2690834e7c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41a89e4fc074253911b2690834e7c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40209 } } }
I20260929 18:57:06.862552 17833 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:06.863137 17837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:06.863605 17837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 18:57:06 dist-test-slave-q2kh krb5kdc[17745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:57:06.865222 17837 catalog_manager.cc:1347] Generated new cluster ID: ce857f5f58164031a6671a44876dd766
I20260929 18:57:06.865271 17837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:06.869342 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:06.857605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42717 (local address 127.0.0.1:40209)
0929 18:57:06.857793 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:06.857802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:06.857836 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:57:06.858335 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:06.858342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:06.858423 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:06.858593 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:06.859242 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:06.860130 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:06.861099 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:06.861663 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:06.861778 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:06.865090 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:06.865094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:06.865094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:06.865107 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:06.867171 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:06.867673 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:06.867676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:06.867676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:06.867779 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:06.868031 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:06.868032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:06.868032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:06.868345 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:06.868510 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:06.869011 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 18:57:06.869113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260929 18:57:06.870625 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:06.879347 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:06.872114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58165 (local address 127.0.0.1:40209)
0929 18:57:06.872149 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:06.872151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:06.872248 (+ 97us) server_negotiation.cc:407] Connection header received
0929 18:57:06.872280 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:06.872281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:06.872290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:06.872354 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:06.872952 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:06.873713 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:06.874702 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:06.875018 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:06.875096 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:06.876470 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:06.876471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:06.876471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:06.876479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:06.878075 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:06.878524 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:06.878526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:06.878526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:06.878586 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:06.878803 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:06.878803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:06.878804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:06.878939 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:06.879046 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:06.879172 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 18:57:06.879244 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:57:06.898859 17837 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:06.899613 17837 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:06.908710 17837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e41a89e4fc074253911b2690834e7c03: Generated new TSK 0
I20260929 18:57:06.908959 17837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:06.914961 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17758
W20260929 18:57:06.919275 17743 connection.cc:535] client connection to 127.0.0.1:40209 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40209): Connection reset by peer (error 104)
2026-09-29T18:57:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:06 dist-test-slave-q2kh krb5kdc[17857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:06 dist-test-slave-q2kh krb5kdc[17857](info): set up 2 sockets
Sep 29 18:57:06 dist-test-slave-q2kh krb5kdc[17857](info): commencing operation
krb5kdc: starting...
W20260929 18:57:07.982116 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:08 dist-test-slave-q2kh krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:08.104729 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:08.184917 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:08.185112 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:08.185133 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:08.186285 17870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:08.186323 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:08.186337 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:08.186350 17870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:08.186362 17870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:08.187800 17870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:08.188040 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:08.188256 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:08.192971 17875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:08.192986 17878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:08.193063 17876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:08.193325 17870 server_base.cc:1034] running on GCE node
I20260929 18:57:08.193555 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:08.193851 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:08.195007 17870 hybrid_clock.cc:648] HybridClock initialized: now 1790708228194984 us; error 38 us; skew 500 ppm
Sep 29 18:57:08 dist-test-slave-q2kh krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:08.202446 17870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:08.202943 17870 webserver.cc:460] Webserver started at http://127.0.0.1:44089/ using document root <none> and password file <none>
I20260929 18:57:08.203174 17870 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:08.203223 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:08.203389 17870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:08.204468 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "8bce0aa1a054421b827f79b2062a2cda"
format_stamp: "Formatted at 2026-09-29 18:57:08 on dist-test-slave-q2kh"
server_key: "0d71636814815539b5049ab062fc57f2"
server_key_iv: "77079be4ea3be3a4c99ea14db9acd12b"
server_key_version: "encryptionkey@0"
I20260929 18:57:08.204834 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "8bce0aa1a054421b827f79b2062a2cda"
format_stamp: "Formatted at 2026-09-29 18:57:08 on dist-test-slave-q2kh"
server_key: "0d71636814815539b5049ab062fc57f2"
server_key_iv: "77079be4ea3be3a4c99ea14db9acd12b"
server_key_version: "encryptionkey@0"
I20260929 18:57:08.206213 17870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 18:57:08.206974 17885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:08.207175 17870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:08.207284 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "8bce0aa1a054421b827f79b2062a2cda"
format_stamp: "Formatted at 2026-09-29 18:57:08 on dist-test-slave-q2kh"
server_key: "0d71636814815539b5049ab062fc57f2"
server_key_iv: "77079be4ea3be3a4c99ea14db9acd12b"
server_key_version: "encryptionkey@0"
I20260929 18:57:08.207353 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:08.214470 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:08.216146 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:08.216300 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:08.220139 17870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38249
I20260929 18:57:08.220208 17937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38249 every 8 connection(s)
I20260929 18:57:08.220541 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:08.221377 17938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:08.223588 17938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda: Bootstrap starting.
I20260929 18:57:08.224107 17938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:08.224326 17938 log.cc:824] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:08.224952 17938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda: No bootstrap required, opened a new log
I20260929 18:57:08.226492 17938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce0aa1a054421b827f79b2062a2cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260929 18:57:08.226624 17938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:08.226646 17938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bce0aa1a054421b827f79b2062a2cda, State: Initialized, Role: FOLLOWER
I20260929 18:57:08.226744 17938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce0aa1a054421b827f79b2062a2cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260929 18:57:08.226822 17938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:08.226859 17938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:08.226941 17938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:08.227458 17938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce0aa1a054421b827f79b2062a2cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260929 18:57:08.227573 17938 leader_election.cc:302] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bce0aa1a054421b827f79b2062a2cda; no voters:
I20260929 18:57:08.227741 17938 leader_election.cc:288] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:08.227777 17941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:08.227962 17941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [term 1 LEADER]: Becoming Leader. State: Replica: 8bce0aa1a054421b827f79b2062a2cda, State: Running, Role: LEADER
I20260929 18:57:08.228000 17938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:08.228080 17941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce0aa1a054421b827f79b2062a2cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } }
I20260929 18:57:08.228462 17943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bce0aa1a054421b827f79b2062a2cda. Latest consensus state: current_term: 1 leader_uuid: "8bce0aa1a054421b827f79b2062a2cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce0aa1a054421b827f79b2062a2cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } } }
I20260929 18:57:08.228582 17942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bce0aa1a054421b827f79b2062a2cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bce0aa1a054421b827f79b2062a2cda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38249 } } }
I20260929 18:57:08.228600 17943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:08.228669 17942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:08.229061 17950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:08.229609 17950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:08.229627 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17870
I20260929 18:57:08.229728 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:08.229930 29982 external_mini_cluster.cc:1428] Setting key 275b49423eab7f139f2eb09a48d67dd8
I20260929 18:57:08.231083 17950 catalog_manager.cc:1347] Generated new cluster ID: cb3cbe01d108493d9533c7b17a5200a6
I20260929 18:57:08.231137 17950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:08.242358 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:08.230524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57363 (local address 127.0.0.1:38249)
0929 18:57:08.230709 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:08.230719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:08.230751 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:57:08.230860 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:08.230867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:08.230923 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:08.238278 (+ 7355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:08.238495 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:08.239577 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:08.240384 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:08.240803 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:08.240928 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:08.241664 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:08.241668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:08.241668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:08.241861 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:08.241890 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:08.241901 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:08.241996 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:08.242077 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 18:57:08.242108 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260929 18:57:08.243317 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:08.246546 17950 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:08.247200 17950 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:08.247886 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:08.244662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55805 (local address 127.0.0.1:38249)
0929 18:57:08.244700 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:08.244702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:08.244821 (+ 119us) server_negotiation.cc:407] Connection header received
0929 18:57:08.244880 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:08.244880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:08.244892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:08.244947 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:08.245101 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:08.245636 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:08.246636 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:08.246868 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:08.246900 (+ 32us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:08.247634 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:08.247635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:08.247635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:08.247640 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:08.247654 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:08.247658 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:08.247705 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:08.247781 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:57:08.247793 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:08.256120 17950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bce0aa1a054421b827f79b2062a2cda: Generated new TSK 0
I20260929 18:57:08.256311 17950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:08.260207 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17870
W20260929 18:57:08.264642 17855 connection.cc:535] client connection to 127.0.0.1:38249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38249): Connection reset by peer (error 104)
2026-09-29T18:57:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:08 dist-test-slave-q2kh krb5kdc[17969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:08 dist-test-slave-q2kh krb5kdc[17969](info): set up 2 sockets
Sep 29 18:57:08 dist-test-slave-q2kh krb5kdc[17969](info): commencing operation
krb5kdc: starting...
W20260929 18:57:09.298226 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:09 dist-test-slave-q2kh krb5kdc[17969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:09.416405 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:09.496677 17982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:09.496953 17982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:09.496995 17982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:09.498232 17982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:09.498303 17982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:09.498327 17982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:09.498347 17982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:09.498378 17982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:09.499881 17982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:09.500152 17982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:09.500382 17982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:09.505267 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:09.505267 17990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:09.505496 17982 server_base.cc:1034] running on GCE node
W20260929 18:57:09.505294 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:09.505784 17982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:09.506090 17982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:09.507290 17982 hybrid_clock.cc:648] HybridClock initialized: now 1790708229507256 us; error 52 us; skew 500 ppm
Sep 29 18:57:09 dist-test-slave-q2kh krb5kdc[17969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:09.509953 17982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:09.510497 17982 webserver.cc:460] Webserver started at http://127.0.0.1:43531/ using document root <none> and password file <none>
I20260929 18:57:09.510733 17982 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:09.510805 17982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:09.510967 17982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:09.512094 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "56c9cc91ee334b7d994b2ee09fc42ad6"
format_stamp: "Formatted at 2026-09-29 18:57:09 on dist-test-slave-q2kh"
server_key: "8341aebfa6823fb8689d4e4bca2b0c67"
server_key_iv: "4eef4a0854bd8a972c74ed3fe2f446a4"
server_key_version: "encryptionkey@0"
I20260929 18:57:09.512483 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "56c9cc91ee334b7d994b2ee09fc42ad6"
format_stamp: "Formatted at 2026-09-29 18:57:09 on dist-test-slave-q2kh"
server_key: "8341aebfa6823fb8689d4e4bca2b0c67"
server_key_iv: "4eef4a0854bd8a972c74ed3fe2f446a4"
server_key_version: "encryptionkey@0"
I20260929 18:57:09.513975 17982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:57:09.514843 17997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:09.515024 17982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:09.515125 17982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "56c9cc91ee334b7d994b2ee09fc42ad6"
format_stamp: "Formatted at 2026-09-29 18:57:09 on dist-test-slave-q2kh"
server_key: "8341aebfa6823fb8689d4e4bca2b0c67"
server_key_iv: "4eef4a0854bd8a972c74ed3fe2f446a4"
server_key_version: "encryptionkey@0"
I20260929 18:57:09.515213 17982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:09.518643 17982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:09.521000 17982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:09.521160 17982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:09.525157 17982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36111
I20260929 18:57:09.525189 18049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36111 every 8 connection(s)
I20260929 18:57:09.525522 17982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:09.526544 18050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:09.528785 18050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6: Bootstrap starting.
I20260929 18:57:09.529369 18050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:09.529654 18050 log.cc:824] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:09.530409 18050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6: No bootstrap required, opened a new log
I20260929 18:57:09.531126 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 17982
I20260929 18:57:09.531244 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:09.531443 29982 external_mini_cluster.cc:1428] Setting key a96b84958ca8159242b76461e001264d
I20260929 18:57:09.532378 18050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 18:57:09.532580 18050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:09.532621 18050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c9cc91ee334b7d994b2ee09fc42ad6, State: Initialized, Role: FOLLOWER
I20260929 18:57:09.532796 18050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 18:57:09.532902 18050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:09.532943 18050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:09.532991 18050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:09.533685 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:09.531948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42343 (local address 127.0.0.1:36111)
0929 18:57:09.532121 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:09.532128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:09.532152 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:57:09.532341 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:09.532346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:09.532369 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:09.532474 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:09.533220 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:09.533224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:09.533224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:09.533229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:09.533278 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:09.533289 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:09.533376 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:09.533448 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:57:09.533490 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260929 18:57:09.533890 18050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 18:57:09.534008 18050 leader_election.cc:302] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56c9cc91ee334b7d994b2ee09fc42ad6; no voters:
I20260929 18:57:09.534183 18050 leader_election.cc:288] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:09.534200 18055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:09.534313 18055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 56c9cc91ee334b7d994b2ee09fc42ad6, State: Running, Role: LEADER
I20260929 18:57:09.534384 18055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 18:57:09.534438 18050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:09.534564 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:09.534688 18057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } } }
I20260929 18:57:09.534755 18057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:09.534937 18056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56c9cc91ee334b7d994b2ee09fc42ad6. Latest consensus state: current_term: 1 leader_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56c9cc91ee334b7d994b2ee09fc42ad6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } } }
I20260929 18:57:09.534983 18056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:09.535044 18060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:09.535583 18060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:09.537281 18060 catalog_manager.cc:1347] Generated new cluster ID: 2ae5b306b4564ad6af5e01a55b72aa4d
I20260929 18:57:09.537345 18060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:09.538178 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:09.536906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48597 (local address 127.0.0.1:36111)
0929 18:57:09.536974 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:09.536976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:09.537096 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:57:09.537170 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:09.537171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:09.537176 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:09.537228 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:09.537909 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:09.537911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:09.537911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:09.537916 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:09.537930 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:09.537934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:09.537984 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:09.538061 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:57:09.538076 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:57:09.547694 18060 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:09.547961 18060 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:09.556831 18060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56c9cc91ee334b7d994b2ee09fc42ad6: Generated new TSK 0
I20260929 18:57:09.557077 18060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:09.567886 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 17982
2026-09-29T18:57:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:09 dist-test-slave-q2kh krb5kdc[18081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:09 dist-test-slave-q2kh krb5kdc[18081](info): set up 2 sockets
Sep 29 18:57:09 dist-test-slave-q2kh krb5kdc[18081](info): commencing operation
krb5kdc: starting...
W20260929 18:57:10.606391 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:10 dist-test-slave-q2kh krb5kdc[18081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:10.732640 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:10.813961 18094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:10.814181 18094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:10.814206 18094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:10.815368 18094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:10.815410 18094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:10.815425 18094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:10.815438 18094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:10.815449 18094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:10.817004 18094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:10.817252 18094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:10.817481 18094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:10.822698 18102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:10.822698 18099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:10.822716 18100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:10.822777 18094 server_base.cc:1034] running on GCE node
I20260929 18:57:10.823261 18094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:10.823524 18094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:10.824690 18094 hybrid_clock.cc:648] HybridClock initialized: now 1790708230824639 us; error 62 us; skew 500 ppm
Sep 29 18:57:10 dist-test-slave-q2kh krb5kdc[18081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:10.827368 18094 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:10.827842 18094 webserver.cc:460] Webserver started at http://127.0.0.1:38487/ using document root <none> and password file <none>
I20260929 18:57:10.828116 18094 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:10.828197 18094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:10.828357 18094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:10.829468 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f"
format_stamp: "Formatted at 2026-09-29 18:57:10 on dist-test-slave-q2kh"
server_key: "8c2707287403486ca6f80e9e941bf1b7"
server_key_iv: "7dcfbfa7e27e85a70f6fe817ac95fd7a"
server_key_version: "encryptionkey@0"
I20260929 18:57:10.829865 18094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f"
format_stamp: "Formatted at 2026-09-29 18:57:10 on dist-test-slave-q2kh"
server_key: "8c2707287403486ca6f80e9e941bf1b7"
server_key_iv: "7dcfbfa7e27e85a70f6fe817ac95fd7a"
server_key_version: "encryptionkey@0"
I20260929 18:57:10.831331 18094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260929 18:57:10.832162 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:10.832332 18094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:10.832430 18094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f"
format_stamp: "Formatted at 2026-09-29 18:57:10 on dist-test-slave-q2kh"
server_key: "8c2707287403486ca6f80e9e941bf1b7"
server_key_iv: "7dcfbfa7e27e85a70f6fe817ac95fd7a"
server_key_version: "encryptionkey@0"
I20260929 18:57:10.832521 18094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:10.872860 18094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:10.875626 18094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:10.875833 18094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:10.880282 18094 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44375
I20260929 18:57:10.880316 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44375 every 8 connection(s)
I20260929 18:57:10.880715 18094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:10.881666 18162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:10.884014 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f: Bootstrap starting.
I20260929 18:57:10.884734 18162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:10.885031 18162 log.cc:824] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:10.885803 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f: No bootstrap required, opened a new log
I20260929 18:57:10.887501 18162 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44375 } }
I20260929 18:57:10.887636 18162 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:10.887658 18162 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5fce8c11bf441f2aa2eed13d0f9d05f, State: Initialized, Role: FOLLOWER
I20260929 18:57:10.887828 18162 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44375 } }
I20260929 18:57:10.887915 18162 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:10.887943 18162 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:10.888006 18162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:10.888011 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18094
I20260929 18:57:10.888136 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:10.888329 29982 external_mini_cluster.cc:1428] Setting key a60d2d025e2962468cd224b4be31db9d
I20260929 18:57:10.888689 18162 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44375 } }
I20260929 18:57:10.888794 18162 leader_election.cc:302] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5fce8c11bf441f2aa2eed13d0f9d05f; no voters:
I20260929 18:57:10.888968 18162 leader_election.cc:288] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:10.889014 18166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:10.889147 18166 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [term 1 LEADER]: Becoming Leader. State: Replica: b5fce8c11bf441f2aa2eed13d0f9d05f, State: Running, Role: LEADER
I20260929 18:57:10.889226 18162 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:10.889262 18166 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44375 } }
I20260929 18:57:10.889602 18166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5fce8c11bf441f2aa2eed13d0f9d05f. Latest consensus state: current_term: 1 leader_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44375 } } }
I20260929 18:57:10.889703 18166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:10.890506 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5fce8c11bf441f2aa2eed13d0f9d05f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44375 } } }
I20260929 18:57:10.890702 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:10.890878 18171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:10.891399 18171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:10.893096 18171 catalog_manager.cc:1347] Generated new cluster ID: 9b04de7d1f76485badd69d4ee109b59c
I20260929 18:57:10.893146 18171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:57:10 dist-test-slave-q2kh krb5kdc[18081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:57:10.897634 18171 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:10.898121 18171 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:10.900856 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:10.888853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58107 (local address 127.0.0.1:44375)
0929 18:57:10.889151 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:10.889161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:10.889196 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:57:10.889313 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:10.889321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:10.889391 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:10.889577 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:10.890498 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:10.891935 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:10.892979 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:10.893569 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:10.893668 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:10.896560 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:10.896564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:10.896564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:10.896577 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:10.899208 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:10.899612 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:10.899616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:10.899616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:10.899733 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:10.899937 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:10.899938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:10.899939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:10.900227 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:10.900390 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:10.900462 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:57:10.900619 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260929 18:57:10.901763 18171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5fce8c11bf441f2aa2eed13d0f9d05f: Generated new TSK 0
I20260929 18:57:10.901881 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:10.903179 18171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:10.909790 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:10.903284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55417 (local address 127.0.0.1:44375)
0929 18:57:10.903318 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:10.903320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:10.903571 (+ 251us) server_negotiation.cc:407] Connection header received
0929 18:57:10.903609 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:10.903610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:10.903620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:10.903676 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:10.904146 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:10.904934 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:10.905645 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:10.905988 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:10.906058 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:10.907245 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:10.907246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:10.907246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:10.907253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:10.908708 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:10.909019 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:10.909020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:10.909021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:10.909080 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:10.909250 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:10.909250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:10.909251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:10.909373 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:10.909468 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:10.909610 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 18:57:10.909689 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:57:10.911952 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18094
2026-09-29T18:57:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:10 dist-test-slave-q2kh krb5kdc[18193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:10 dist-test-slave-q2kh krb5kdc[18193](info): set up 2 sockets
Sep 29 18:57:10 dist-test-slave-q2kh krb5kdc[18193](info): commencing operation
krb5kdc: starting...
W20260929 18:57:11.962505 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:12 dist-test-slave-q2kh krb5kdc[18193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:12.085135 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:12.165113 18206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:12.165309 18206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:12.165333 18206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:12.166411 18206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:12.166446 18206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:12.166460 18206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:12.166472 18206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:12.166484 18206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:12.167860 18206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:12.168036 18206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:12.168202 18206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:12.172909 18214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:12.172901 18211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:12.173100 18206 server_base.cc:1034] running on GCE node
W20260929 18:57:12.172921 18212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:12.173429 18206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:12.173717 18206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:12.174893 18206 hybrid_clock.cc:648] HybridClock initialized: now 1790708232174869 us; error 35 us; skew 500 ppm
Sep 29 18:57:12 dist-test-slave-q2kh krb5kdc[18193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:12.177332 18206 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:12.177749 18206 webserver.cc:460] Webserver started at http://127.0.0.1:37227/ using document root <none> and password file <none>
I20260929 18:57:12.177996 18206 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:12.178066 18206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:12.178257 18206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:12.179772 18206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "8005041ff50043238e58b04418051b58"
format_stamp: "Formatted at 2026-09-29 18:57:12 on dist-test-slave-q2kh"
server_key: "a5f565f84671673c1a0c55398f16d696"
server_key_iv: "4e4463f53434156a8763bc134dc2bb72"
server_key_version: "encryptionkey@0"
I20260929 18:57:12.180142 18206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "8005041ff50043238e58b04418051b58"
format_stamp: "Formatted at 2026-09-29 18:57:12 on dist-test-slave-q2kh"
server_key: "a5f565f84671673c1a0c55398f16d696"
server_key_iv: "4e4463f53434156a8763bc134dc2bb72"
server_key_version: "encryptionkey@0"
I20260929 18:57:12.181491 18206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:12.182312 18221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:12.182511 18206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:12.182610 18206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "8005041ff50043238e58b04418051b58"
format_stamp: "Formatted at 2026-09-29 18:57:12 on dist-test-slave-q2kh"
server_key: "a5f565f84671673c1a0c55398f16d696"
server_key_iv: "4e4463f53434156a8763bc134dc2bb72"
server_key_version: "encryptionkey@0"
I20260929 18:57:12.182695 18206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:12.198671 18206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:12.200757 18206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:12.200975 18206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:12.205458 18206 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40119
I20260929 18:57:12.205490 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40119 every 8 connection(s)
I20260929 18:57:12.205885 18206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:12.206791 18274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:12.208973 18274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58: Bootstrap starting.
I20260929 18:57:12.209483 18274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:12.209733 18274 log.cc:824] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:12.209853 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18206
I20260929 18:57:12.209993 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:12.210237 29982 external_mini_cluster.cc:1428] Setting key 8fdf4fd26c5b4d1630267f13a53cfcbc
I20260929 18:57:12.210587 18274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58: No bootstrap required, opened a new log
I20260929 18:57:12.213341 18274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8005041ff50043238e58b04418051b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40119 } }
I20260929 18:57:12.213505 18274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:12.213533 18274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8005041ff50043238e58b04418051b58, State: Initialized, Role: FOLLOWER
I20260929 18:57:12.213644 18274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8005041ff50043238e58b04418051b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40119 } }
I20260929 18:57:12.213713 18274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:12.213747 18274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:12.213853 18274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:12.214478 18274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8005041ff50043238e58b04418051b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40119 } }
I20260929 18:57:12.214597 18274 leader_election.cc:302] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8005041ff50043238e58b04418051b58; no voters:
I20260929 18:57:12.214772 18274 leader_election.cc:288] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:12.214793 18279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:12.214913 18279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [term 1 LEADER]: Becoming Leader. State: Replica: 8005041ff50043238e58b04418051b58, State: Running, Role: LEADER
I20260929 18:57:12.215045 18274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:12.215041 18279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8005041ff50043238e58b04418051b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40119 } }
I20260929 18:57:12.215497 18280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8005041ff50043238e58b04418051b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8005041ff50043238e58b04418051b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40119 } } }
I20260929 18:57:12.215636 18281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8005041ff50043238e58b04418051b58. Latest consensus state: current_term: 1 leader_uuid: "8005041ff50043238e58b04418051b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8005041ff50043238e58b04418051b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40119 } } }
I20260929 18:57:12.215703 18280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:12.215806 18281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:12.216848 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:12.210806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33157 (local address 127.0.0.1:40119)
0929 18:57:12.210997 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:12.211006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:12.211042 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:57:12.211879 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:12.211888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:12.211952 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:12.212103 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:12.212267 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:12.213688 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:12.214621 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:12.215198 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:12.215349 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:12.216118 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:12.216122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:12.216123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:12.216307 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:12.216342 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:12.216352 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:12.216401 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:12.216542 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 18:57:12.216615 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
W20260929 18:57:12.217167 18295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:12.217236 18295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:12.217326 18284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:12.217519 18284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:12.217885 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:12.219111 18284 catalog_manager.cc:1347] Generated new cluster ID: d92b74a51a554a76b1e3daee99c52f64
I20260929 18:57:12.219166 18284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:12.222729 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:12.219359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59417 (local address 127.0.0.1:40119)
0929 18:57:12.219391 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:12.219393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:12.219513 (+ 120us) server_negotiation.cc:407] Connection header received
0929 18:57:12.219539 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:12.219540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:12.219550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:12.219609 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:12.219745 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:12.220463 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:12.221411 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:12.221722 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:12.221795 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:12.222491 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:12.222493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:12.222493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:12.222497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:12.222514 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:12.222519 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:12.222577 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:12.222628 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 18:57:12.222650 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 18:57:12.231426 18284 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:12.232177 18284 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:12.248018 18284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8005041ff50043238e58b04418051b58: Generated new TSK 0
I20260929 18:57:12.248286 18284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:12.253695 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18206
2026-09-29T18:57:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:12 dist-test-slave-q2kh krb5kdc[18305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:12 dist-test-slave-q2kh krb5kdc[18305](info): set up 2 sockets
Sep 29 18:57:12 dist-test-slave-q2kh krb5kdc[18305](info): commencing operation
krb5kdc: starting...
W20260929 18:57:13.278370 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:13 dist-test-slave-q2kh krb5kdc[18305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:13.396739 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:13.476102 18318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:13.476298 18318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:13.476320 18318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:13.477479 18318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:13.477520 18318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:13.477535 18318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:13.477546 18318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:13.477595 18318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:13.479103 18318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:13.479348 18318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:13.479575 18318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:13.484357 18323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:13.484381 18324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:13.484385 18326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:13.484535 18318 server_base.cc:1034] running on GCE node
I20260929 18:57:13.484781 18318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:13.485071 18318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:13.486227 18318 hybrid_clock.cc:648] HybridClock initialized: now 1790708233486205 us; error 32 us; skew 500 ppm
Sep 29 18:57:13 dist-test-slave-q2kh krb5kdc[18305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:13.495028 18318 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:13.495412 18318 webserver.cc:460] Webserver started at http://127.0.0.1:40917/ using document root <none> and password file <none>
I20260929 18:57:13.495602 18318 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:13.495640 18318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:13.495723 18318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:13.496742 18318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "2727dce1f17a47f58532310aacf1567c"
format_stamp: "Formatted at 2026-09-29 18:57:13 on dist-test-slave-q2kh"
server_key: "397078d58c6a1bf0d034f39b6f1a18d0"
server_key_iv: "995114b684c022e71a28067f2e8ceba9"
server_key_version: "encryptionkey@0"
I20260929 18:57:13.497035 18318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "2727dce1f17a47f58532310aacf1567c"
format_stamp: "Formatted at 2026-09-29 18:57:13 on dist-test-slave-q2kh"
server_key: "397078d58c6a1bf0d034f39b6f1a18d0"
server_key_iv: "995114b684c022e71a28067f2e8ceba9"
server_key_version: "encryptionkey@0"
I20260929 18:57:13.498289 18318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:13.498991 18333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:13.499162 18318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:13.499250 18318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "2727dce1f17a47f58532310aacf1567c"
format_stamp: "Formatted at 2026-09-29 18:57:13 on dist-test-slave-q2kh"
server_key: "397078d58c6a1bf0d034f39b6f1a18d0"
server_key_iv: "995114b684c022e71a28067f2e8ceba9"
server_key_version: "encryptionkey@0"
I20260929 18:57:13.499306 18318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:13.502480 18318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:13.504251 18318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:13.504351 18318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:13.508081 18318 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33509
I20260929 18:57:13.508108 18385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33509 every 8 connection(s)
I20260929 18:57:13.508430 18318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:13.509418 18386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:13.511036 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18318
I20260929 18:57:13.511145 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:13.511320 29982 external_mini_cluster.cc:1428] Setting key 135a52ffa64031dafa1ed9b1453032fa
I20260929 18:57:13.512020 18386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c: Bootstrap starting.
I20260929 18:57:13.512681 18386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:13.512995 18386 log.cc:824] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:13.513823 18386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c: No bootstrap required, opened a new log
I20260929 18:57:13.513991 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:13.511848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38787 (local address 127.0.0.1:33509)
0929 18:57:13.512032 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:13.512041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:13.512072 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:57:13.512669 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:13.512678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:13.512713 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:13.512877 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:13.513497 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:13.513500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:13.513500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:13.513504 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:13.513549 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:13.513559 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:13.513629 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:13.513704 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:57:13.513753 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260929 18:57:13.514977 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:13.516179 18386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2727dce1f17a47f58532310aacf1567c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20260929 18:57:13.516371 18386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:13.516425 18386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2727dce1f17a47f58532310aacf1567c, State: Initialized, Role: FOLLOWER
I20260929 18:57:13.516546 18386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2727dce1f17a47f58532310aacf1567c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20260929 18:57:13.516678 18386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:13.516731 18386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:13.516773 18386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:13.517489 18386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2727dce1f17a47f58532310aacf1567c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20260929 18:57:13.517601 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:13.516572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60477 (local address 127.0.0.1:33509)
0929 18:57:13.516628 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:13.516630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:13.516700 (+ 70us) server_negotiation.cc:407] Connection header received
0929 18:57:13.516787 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:13.516787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:13.516792 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:13.516871 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:13.517411 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:13.517412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:13.517412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:13.517417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:13.517432 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:13.517436 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:13.517502 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:13.517538 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 18:57:13.517551 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:57:13.517606 18386 leader_election.cc:302] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2727dce1f17a47f58532310aacf1567c; no voters:
I20260929 18:57:13.517791 18386 leader_election.cc:288] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:13.517812 18396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:13.518042 18386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:13.518055 18396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [term 1 LEADER]: Becoming Leader. State: Replica: 2727dce1f17a47f58532310aacf1567c, State: Running, Role: LEADER
I20260929 18:57:13.518224 18396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2727dce1f17a47f58532310aacf1567c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } }
I20260929 18:57:13.518524 18397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2727dce1f17a47f58532310aacf1567c. Latest consensus state: current_term: 1 leader_uuid: "2727dce1f17a47f58532310aacf1567c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2727dce1f17a47f58532310aacf1567c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } } }
I20260929 18:57:13.518524 18398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2727dce1f17a47f58532310aacf1567c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2727dce1f17a47f58532310aacf1567c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33509 } } }
I20260929 18:57:13.518693 18397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:13.518704 18398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c [sys.catalog]: This master's current role is: LEADER
W20260929 18:57:13.520119 18411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:13.520184 18411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:13.520239 18412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:13.520421 18412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:13.521710 18412 catalog_manager.cc:1347] Generated new cluster ID: 254d6241f5be4638a1a47ca26241304d
I20260929 18:57:13.521759 18412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:13.529772 18412 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:13.529956 18412 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:13.537266 18412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2727dce1f17a47f58532310aacf1567c: Generated new TSK 0
I20260929 18:57:13.537456 18412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:13.541692 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18318
2026-09-29T18:57:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:13 dist-test-slave-q2kh krb5kdc[18417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:13 dist-test-slave-q2kh krb5kdc[18417](info): set up 2 sockets
Sep 29 18:57:13 dist-test-slave-q2kh krb5kdc[18417](info): commencing operation
krb5kdc: starting...
W20260929 18:57:14.570294 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:14 dist-test-slave-q2kh krb5kdc[18417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:14.692698 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:14.775444 18430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:14.775666 18430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:14.775702 18430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:14.776847 18430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:14.776914 18430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:14.776938 18430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:14.776964 18430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:14.776996 18430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:14.778466 18430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:14.778715 18430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:14.778947 18430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:14.783844 18436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:14.783905 18438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:14.783917 18430 server_base.cc:1034] running on GCE node
W20260929 18:57:14.783847 18435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:14.784314 18430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:14.784600 18430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:14.785741 18430 hybrid_clock.cc:648] HybridClock initialized: now 1790708234785727 us; error 29 us; skew 500 ppm
Sep 29 18:57:14 dist-test-slave-q2kh krb5kdc[18417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:14.788200 18430 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:14.788659 18430 webserver.cc:460] Webserver started at http://127.0.0.1:42793/ using document root <none> and password file <none>
I20260929 18:57:14.788890 18430 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:14.788933 18430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:14.789094 18430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:14.790112 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "52c0f55a1997443ebe7b859474a7a670"
format_stamp: "Formatted at 2026-09-29 18:57:14 on dist-test-slave-q2kh"
server_key: "6938bdb6dad86d6052d7cf1de05d8a00"
server_key_iv: "2b6f943e5dd6cddde8452636e0a1737e"
server_key_version: "encryptionkey@0"
I20260929 18:57:14.790535 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "52c0f55a1997443ebe7b859474a7a670"
format_stamp: "Formatted at 2026-09-29 18:57:14 on dist-test-slave-q2kh"
server_key: "6938bdb6dad86d6052d7cf1de05d8a00"
server_key_iv: "2b6f943e5dd6cddde8452636e0a1737e"
server_key_version: "encryptionkey@0"
I20260929 18:57:14.791800 18430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:14.792507 18445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:14.792691 18430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:14.792779 18430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "52c0f55a1997443ebe7b859474a7a670"
format_stamp: "Formatted at 2026-09-29 18:57:14 on dist-test-slave-q2kh"
server_key: "6938bdb6dad86d6052d7cf1de05d8a00"
server_key_iv: "2b6f943e5dd6cddde8452636e0a1737e"
server_key_version: "encryptionkey@0"
I20260929 18:57:14.792850 18430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:14.816068 18430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:14.818518 18430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:14.818684 18430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:14.822630 18430 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45635
I20260929 18:57:14.822693 18497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45635 every 8 connection(s)
I20260929 18:57:14.823060 18430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:14.823900 18498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:14.826108 18498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670: Bootstrap starting.
I20260929 18:57:14.826707 18498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:14.826994 18498 log.cc:824] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:14.827350 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18430
I20260929 18:57:14.827471 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:14.827670 29982 external_mini_cluster.cc:1428] Setting key 4312979cf0f2474a78fde537ca77a02a
I20260929 18:57:14.827793 18498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670: No bootstrap required, opened a new log
I20260929 18:57:14.830217 18498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0f55a1997443ebe7b859474a7a670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260929 18:57:14.830370 18498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:14.830395 18498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c0f55a1997443ebe7b859474a7a670, State: Initialized, Role: FOLLOWER
I20260929 18:57:14.830520 18498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0f55a1997443ebe7b859474a7a670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260929 18:57:14.830592 18498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:14.830614 18498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:14.830678 18498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:14.831207 18498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0f55a1997443ebe7b859474a7a670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260929 18:57:14.831287 18498 leader_election.cc:302] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c0f55a1997443ebe7b859474a7a670; no voters:
I20260929 18:57:14.831463 18498 leader_election.cc:288] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:14.831483 18503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:14.831593 18503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [term 1 LEADER]: Becoming Leader. State: Replica: 52c0f55a1997443ebe7b859474a7a670, State: Running, Role: LEADER
I20260929 18:57:14.831705 18498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:14.831728 18503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0f55a1997443ebe7b859474a7a670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260929 18:57:14.832221 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c0f55a1997443ebe7b859474a7a670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0f55a1997443ebe7b859474a7a670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } } }
I20260929 18:57:14.832311 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:14.832463 18505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c0f55a1997443ebe7b859474a7a670. Latest consensus state: current_term: 1 leader_uuid: "52c0f55a1997443ebe7b859474a7a670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c0f55a1997443ebe7b859474a7a670" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } } }
I20260929 18:57:14.832513 18505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:14.832752 18512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:14.833343 18512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:14.835035 18512 catalog_manager.cc:1347] Generated new cluster ID: 0c6afc5eee0349b8b009ee5ee4d779a0
I20260929 18:57:14.835095 18512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:57:14 dist-test-slave-q2kh krb5kdc[18417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:57:14.840453 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:14.828194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:40873 (local address 127.0.0.1:45635)
0929 18:57:14.828372 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:14.828380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:14.828407 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:57:14.829094 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:14.829101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:14.829152 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:14.829264 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:14.830040 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:14.831479 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:14.832514 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:14.833117 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:14.833642 (+ 525us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:14.836774 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:14.836778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:14.836779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:14.836791 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:14.838739 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:14.839205 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:14.839208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:14.839208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:14.839311 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:14.839561 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:14.839562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:14.839562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:14.839857 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:14.840015 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:14.840159 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:57:14.840255 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260929 18:57:14.841504 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:14.847849 18512 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:14.848614 18512 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:14.850358 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:14.842957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:50011 (local address 127.0.0.1:45635)
0929 18:57:14.842991 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:14.842993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:14.843111 (+ 118us) server_negotiation.cc:407] Connection header received
0929 18:57:14.843170 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:14.843171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:14.843180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:14.843236 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:14.843863 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:14.844652 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:14.845688 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:14.846019 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:14.846081 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:14.847616 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:14.847617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:14.847617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:14.847624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:14.849141 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:14.849519 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:14.849520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:14.849521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:14.849577 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:14.849808 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:14.849810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:14.849810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:14.849938 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:14.850053 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:14.850200 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:57:14.850263 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 18:57:14.863720 18512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c0f55a1997443ebe7b859474a7a670: Generated new TSK 0
I20260929 18:57:14.863922 18512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:14.868449 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18430
W20260929 18:57:14.872839 18415 connection.cc:535] client connection to 127.0.0.1:45635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45635): Connection reset by peer (error 104)
2026-09-29T18:57:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:14 dist-test-slave-q2kh krb5kdc[18529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:14 dist-test-slave-q2kh krb5kdc[18529](info): set up 2 sockets
Sep 29 18:57:14 dist-test-slave-q2kh krb5kdc[18529](info): commencing operation
krb5kdc: starting...
W20260929 18:57:15.930260 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:16 dist-test-slave-q2kh krb5kdc[18529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:16.050462 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:16.129662 18542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:16.129915 18542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:16.129951 18542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:16.131388 18542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:16.131479 18542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:16.131506 18542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:16.131531 18542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:16.131558 18542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:16.133261 18542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:16.133553 18542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:16.133803 18542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:16.138695 18547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:16.138756 18550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:16.138859 18548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:16.138792 18542 server_base.cc:1034] running on GCE node
I20260929 18:57:16.139171 18542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:16.139459 18542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:16.140633 18542 hybrid_clock.cc:648] HybridClock initialized: now 1790708236140616 us; error 48 us; skew 500 ppm
Sep 29 18:57:16 dist-test-slave-q2kh krb5kdc[18529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:16.143079 18542 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:16.143546 18542 webserver.cc:460] Webserver started at http://127.0.0.1:40739/ using document root <none> and password file <none>
I20260929 18:57:16.143769 18542 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:16.143837 18542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:16.143992 18542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:16.145045 18542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "58adf51193644facb2b43d22a2a18419"
format_stamp: "Formatted at 2026-09-29 18:57:16 on dist-test-slave-q2kh"
server_key: "a43be29b0df067b397a3d1b910486288"
server_key_iv: "fb2f30969f40b04645be66275f981952"
server_key_version: "encryptionkey@0"
I20260929 18:57:16.145407 18542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "58adf51193644facb2b43d22a2a18419"
format_stamp: "Formatted at 2026-09-29 18:57:16 on dist-test-slave-q2kh"
server_key: "a43be29b0df067b397a3d1b910486288"
server_key_iv: "fb2f30969f40b04645be66275f981952"
server_key_version: "encryptionkey@0"
I20260929 18:57:16.146788 18542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:16.147522 18557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:16.147702 18542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:16.147801 18542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "58adf51193644facb2b43d22a2a18419"
format_stamp: "Formatted at 2026-09-29 18:57:16 on dist-test-slave-q2kh"
server_key: "a43be29b0df067b397a3d1b910486288"
server_key_iv: "fb2f30969f40b04645be66275f981952"
server_key_version: "encryptionkey@0"
I20260929 18:57:16.147897 18542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:16.165530 18542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:16.167502 18542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:16.167670 18542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:16.171653 18542 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41355
I20260929 18:57:16.171685 18609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41355 every 8 connection(s)
I20260929 18:57:16.172035 18542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:16.172889 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:16.175024 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18542
I20260929 18:57:16.175153 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:16.175339 29982 external_mini_cluster.cc:1428] Setting key 8e11c8b127da4d99bd89fb933a6248a2
I20260929 18:57:16.175561 18610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419: Bootstrap starting.
I20260929 18:57:16.176268 18610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:16.176582 18610 log.cc:824] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:16.177309 18610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419: No bootstrap required, opened a new log
I20260929 18:57:16.179517 18610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58adf51193644facb2b43d22a2a18419" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 18:57:16.179700 18610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:16.179741 18610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58adf51193644facb2b43d22a2a18419, State: Initialized, Role: FOLLOWER
I20260929 18:57:16.179937 18610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58adf51193644facb2b43d22a2a18419" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 18:57:16.180024 18610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:16.180047 18610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:16.180114 18610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:16.180851 18610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58adf51193644facb2b43d22a2a18419" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 18:57:16.180969 18610 leader_election.cc:302] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58adf51193644facb2b43d22a2a18419; no voters:
I20260929 18:57:16.181149 18610 leader_election.cc:288] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:16.181181 18615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:16.181386 18615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [term 1 LEADER]: Becoming Leader. State: Replica: 58adf51193644facb2b43d22a2a18419, State: Running, Role: LEADER
I20260929 18:57:16.181425 18610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:16.181505 18615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58adf51193644facb2b43d22a2a18419" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260929 18:57:16.181228 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:16.175860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:42527 (local address 127.0.0.1:41355)
0929 18:57:16.176034 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:16.176043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:16.176080 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:57:16.176577 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:16.176584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:16.176658 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:16.176825 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:16.177010 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:16.178002 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:16.179178 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:16.179573 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:16.179659 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:16.180517 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:16.180521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:16.180521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:16.180691 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:16.180721 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:16.180732 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:16.180797 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:16.180975 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:57:16.181009 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260929 18:57:16.182013 18617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58adf51193644facb2b43d22a2a18419. Latest consensus state: current_term: 1 leader_uuid: "58adf51193644facb2b43d22a2a18419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58adf51193644facb2b43d22a2a18419" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260929 18:57:16.182184 18617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:16.181902 18616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58adf51193644facb2b43d22a2a18419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58adf51193644facb2b43d22a2a18419" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260929 18:57:16.182317 18616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:16.183030 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:57:16.183403 18631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:16.183445 18631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:16.183506 18624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:16.183673 18624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:16.185041 18624 catalog_manager.cc:1347] Generated new cluster ID: e76f72d094694351bd607adfacccb9ba
I20260929 18:57:16.185115 18624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:16.190092 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:16.186526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:37633 (local address 127.0.0.1:41355)
0929 18:57:16.186560 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:16.186561 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:16.186693 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:57:16.186724 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:16.186725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:16.186735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:16.186798 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:16.186943 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:16.187706 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:16.188680 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:16.189062 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:16.189157 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:16.189839 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:16.189840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:16.189840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:16.189845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:16.189867 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:16.189872 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:16.189937 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:16.190010 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 18:57:16.190021 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:57:16.197211 18624 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:16.197942 18624 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:16.204625 18624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58adf51193644facb2b43d22a2a18419: Generated new TSK 0
I20260929 18:57:16.204841 18624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:16.207065 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18542
W20260929 18:57:16.210963 18527 connection.cc:535] client connection to 127.0.0.1:41355 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41355): Connection reset by peer (error 104)
2026-09-29T18:57:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:16 dist-test-slave-q2kh krb5kdc[18641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:16 dist-test-slave-q2kh krb5kdc[18641](info): set up 2 sockets
Sep 29 18:57:16 dist-test-slave-q2kh krb5kdc[18641](info): commencing operation
krb5kdc: starting...
W20260929 18:57:17.330327 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:17 dist-test-slave-q2kh krb5kdc[18641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:17.448879 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:17.527949 18654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:17.528198 18654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:17.528236 18654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:17.529392 18654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:17.529464 18654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:17.529488 18654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:17.529511 18654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:17.529539 18654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:17.531021 18654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:17.531306 18654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:17.531535 18654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:17.536432 18662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:17.536432 18660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:17.536551 18654 server_base.cc:1034] running on GCE node
W20260929 18:57:17.536468 18659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:17.536886 18654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:17.537175 18654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:17.538336 18654 hybrid_clock.cc:648] HybridClock initialized: now 1790708237538337 us; error 44 us; skew 500 ppm
Sep 29 18:57:17 dist-test-slave-q2kh krb5kdc[18641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:17.540681 18654 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:17.541147 18654 webserver.cc:460] Webserver started at http://127.0.0.1:33609/ using document root <none> and password file <none>
I20260929 18:57:17.541371 18654 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:17.541440 18654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:17.541596 18654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:17.542691 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "0eb8cd164bf7415a9adc062d196a95e0"
format_stamp: "Formatted at 2026-09-29 18:57:17 on dist-test-slave-q2kh"
server_key: "8f4b195f28991c2f4472f1395f3c7816"
server_key_iv: "23a8ed107449950c29413cb2e9870323"
server_key_version: "encryptionkey@0"
I20260929 18:57:17.543072 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "0eb8cd164bf7415a9adc062d196a95e0"
format_stamp: "Formatted at 2026-09-29 18:57:17 on dist-test-slave-q2kh"
server_key: "8f4b195f28991c2f4472f1395f3c7816"
server_key_iv: "23a8ed107449950c29413cb2e9870323"
server_key_version: "encryptionkey@0"
I20260929 18:57:17.544348 18654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:57:17.545020 18669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:17.545186 18654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:17.545282 18654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "0eb8cd164bf7415a9adc062d196a95e0"
format_stamp: "Formatted at 2026-09-29 18:57:17 on dist-test-slave-q2kh"
server_key: "8f4b195f28991c2f4472f1395f3c7816"
server_key_iv: "23a8ed107449950c29413cb2e9870323"
server_key_version: "encryptionkey@0"
I20260929 18:57:17.545365 18654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:17.548573 18654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:17.550411 18654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:17.550560 18654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:17.554515 18654 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43813
I20260929 18:57:17.554543 18721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43813 every 8 connection(s)
I20260929 18:57:17.554888 18654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:17.555840 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:17.558089 18722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0: Bootstrap starting.
I20260929 18:57:17.558677 18722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:17.558960 18722 log.cc:824] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:17.559664 18722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0: No bootstrap required, opened a new log
I20260929 18:57:17.561409 18722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8cd164bf7415a9adc062d196a95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43813 } }
I20260929 18:57:17.561542 18722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:17.561563 18722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb8cd164bf7415a9adc062d196a95e0, State: Initialized, Role: FOLLOWER
I20260929 18:57:17.561734 18722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8cd164bf7415a9adc062d196a95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43813 } }
I20260929 18:57:17.561810 18722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:17.561859 18722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:17.561908 18722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:17.562494 18722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8cd164bf7415a9adc062d196a95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43813 } }
I20260929 18:57:17.562578 18722 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb8cd164bf7415a9adc062d196a95e0; no voters:
I20260929 18:57:17.562788 18722 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:17.562865 18725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:17.563055 18725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb8cd164bf7415a9adc062d196a95e0, State: Running, Role: LEADER
I20260929 18:57:17.563058 18722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:17.563342 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18654
I20260929 18:57:17.563463 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:17.563661 29982 external_mini_cluster.cc:1428] Setting key a561337502b336056e58db137516523c
I20260929 18:57:17.563853 18725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8cd164bf7415a9adc062d196a95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43813 } }
I20260929 18:57:17.564535 18726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb8cd164bf7415a9adc062d196a95e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8cd164bf7415a9adc062d196a95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43813 } } }
I20260929 18:57:17.564535 18727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb8cd164bf7415a9adc062d196a95e0. Latest consensus state: current_term: 1 leader_uuid: "0eb8cd164bf7415a9adc062d196a95e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8cd164bf7415a9adc062d196a95e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43813 } } }
I20260929 18:57:17.564817 18726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:17.564838 18727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0 [sys.catalog]: This master's current role is: LEADER
W20260929 18:57:17.565002 18740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:17.565052 18740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:17.565171 18743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:17.565335 18743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:17.567059 18743 catalog_manager.cc:1347] Generated new cluster ID: 8441f78a1bd2411f89895c18887e19de
I20260929 18:57:17.567107 18743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:17.567335 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:17.564387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:55207 (local address 127.0.0.1:43813)
0929 18:57:17.564758 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:17.564766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:17.565429 (+ 663us) server_negotiation.cc:407] Connection header received
0929 18:57:17.565640 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:17.565647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:17.565684 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:17.565795 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:17.566836 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:17.566840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:17.566840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:17.566845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:17.566887 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:17.566902 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:17.566985 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:17.567129 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:57:17.567176 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":86,"threads_started":1}
I20260929 18:57:17.568292 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:17.571426 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:17.569851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:59053 (local address 127.0.0.1:43813)
0929 18:57:17.569890 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:17.569892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:17.570482 (+ 590us) server_negotiation.cc:407] Connection header received
0929 18:57:17.570518 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:17.570519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:17.570525 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:17.570605 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:17.571192 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:17.571193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:17.571193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:17.571198 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:17.571213 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:17.571218 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:17.571287 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:17.571343 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 18:57:17.571357 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:57:17.579602 18743 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:17.579798 18743 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:17.601890 18743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb8cd164bf7415a9adc062d196a95e0: Generated new TSK 0
I20260929 18:57:17.602135 18743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:17.611397 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18654
2026-09-29T18:57:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:17 dist-test-slave-q2kh krb5kdc[18753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:17 dist-test-slave-q2kh krb5kdc[18753](info): set up 2 sockets
Sep 29 18:57:17 dist-test-slave-q2kh krb5kdc[18753](info): commencing operation
krb5kdc: starting...
W20260929 18:57:18.634352 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:18 dist-test-slave-q2kh krb5kdc[18753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:18.757318 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:18.837977 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:18.838274 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:18.838304 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:18.839490 18766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:18.839533 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:18.839547 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:18.839560 18766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:18.839572 18766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:18.841027 18766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:18.841233 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:18.841408 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:18.846225 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:18.846326 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:18.846374 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:18.846679 18766 server_base.cc:1034] running on GCE node
I20260929 18:57:18.846925 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:18.847214 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:18.848380 18766 hybrid_clock.cc:648] HybridClock initialized: now 1790708238848368 us; error 29 us; skew 500 ppm
Sep 29 18:57:18 dist-test-slave-q2kh krb5kdc[18753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:18.856397 18766 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:18.856889 18766 webserver.cc:460] Webserver started at http://127.0.0.1:32965/ using document root <none> and password file <none>
I20260929 18:57:18.857127 18766 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:18.857174 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:18.857307 18766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:18.858498 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "5f32b0f087904e518a37095269dac9bc"
format_stamp: "Formatted at 2026-09-29 18:57:18 on dist-test-slave-q2kh"
server_key: "276c4afc9ce24fb01a33e6ca6d6cad2e"
server_key_iv: "86496c6619448cd65e84b76f7c6ea886"
server_key_version: "encryptionkey@0"
I20260929 18:57:18.858865 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "5f32b0f087904e518a37095269dac9bc"
format_stamp: "Formatted at 2026-09-29 18:57:18 on dist-test-slave-q2kh"
server_key: "276c4afc9ce24fb01a33e6ca6d6cad2e"
server_key_iv: "86496c6619448cd65e84b76f7c6ea886"
server_key_version: "encryptionkey@0"
I20260929 18:57:18.860239 18766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 18:57:18.861012 18781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:18.861255 18766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:18.861330 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "5f32b0f087904e518a37095269dac9bc"
format_stamp: "Formatted at 2026-09-29 18:57:18 on dist-test-slave-q2kh"
server_key: "276c4afc9ce24fb01a33e6ca6d6cad2e"
server_key_iv: "86496c6619448cd65e84b76f7c6ea886"
server_key_version: "encryptionkey@0"
I20260929 18:57:18.861430 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:18.879308 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:18.881242 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:18.881417 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:18.885512 18766 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45017
I20260929 18:57:18.885547 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45017 every 8 connection(s)
I20260929 18:57:18.885918 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:18.886847 18834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:18.889060 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc: Bootstrap starting.
I20260929 18:57:18.889592 18834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:18.889820 18834 log.cc:824] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:18.890519 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc: No bootstrap required, opened a new log
I20260929 18:57:18.892027 18834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f32b0f087904e518a37095269dac9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260929 18:57:18.892215 18834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:18.892257 18834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f32b0f087904e518a37095269dac9bc, State: Initialized, Role: FOLLOWER
I20260929 18:57:18.892313 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18766
I20260929 18:57:18.892417 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:18.892437 18834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f32b0f087904e518a37095269dac9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260929 18:57:18.892529 18834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:18.892576 18834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:18.892608 29982 external_mini_cluster.cc:1428] Setting key 0d4660d6b6c8659a3019cce047468704
I20260929 18:57:18.892619 18834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:18.893308 18834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f32b0f087904e518a37095269dac9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260929 18:57:18.893417 18834 leader_election.cc:302] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f32b0f087904e518a37095269dac9bc; no voters:
I20260929 18:57:18.893608 18834 leader_election.cc:288] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:18.893664 18839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:18.893807 18839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [term 1 LEADER]: Becoming Leader. State: Replica: 5f32b0f087904e518a37095269dac9bc, State: Running, Role: LEADER
I20260929 18:57:18.893857 18834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:18.893934 18839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f32b0f087904e518a37095269dac9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } }
I20260929 18:57:18.894375 18839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f32b0f087904e518a37095269dac9bc. Latest consensus state: current_term: 1 leader_uuid: "5f32b0f087904e518a37095269dac9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f32b0f087904e518a37095269dac9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } } }
I20260929 18:57:18.894382 18840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f32b0f087904e518a37095269dac9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f32b0f087904e518a37095269dac9bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45017 } } }
I20260929 18:57:18.894528 18839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:18.894536 18840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:18.894768 18848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:18.895483 18848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:18.897765 18848 catalog_manager.cc:1347] Generated new cluster ID: 3a1bf4a780a448fcbd67aae2dc7b6105
I20260929 18:57:18.897818 18848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:57:18 dist-test-slave-q2kh krb5kdc[18753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:57:18.904353 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:18.893406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:39265 (local address 127.0.0.1:45017)
0929 18:57:18.893589 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:18.893599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:18.893637 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:57:18.893760 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:18.893768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:18.893838 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:18.894876 (+ 1038us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:18.894999 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:18.896346 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:18.897414 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:18.897986 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:18.898115 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:18.900532 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:18.900536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:18.900536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:18.900556 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:18.902707 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:18.903118 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:18.903121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:18.903122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:18.903226 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:18.903472 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:18.903473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:18.903473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:18.903713 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:18.903891 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:18.904027 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 18:57:18.904135 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260929 18:57:18.905397 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:18.917047 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:18.910549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:49361 (local address 127.0.0.1:45017)
0929 18:57:18.910594 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:18.910596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:18.910786 (+ 190us) server_negotiation.cc:407] Connection header received
0929 18:57:18.910874 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:18.910875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:18.910887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:18.910957 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:18.911505 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:18.912017 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:18.912737 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:18.912953 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:18.913044 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:18.914373 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:18.914375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:18.914375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:18.914382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:18.915800 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:18.916181 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:18.916182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:18.916183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:18.916248 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:18.916488 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:18.916488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:18.916489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:18.916600 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:18.916723 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:18.916867 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:57:18.916931 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:57:18.920045 18848 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:18.920583 18848 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:18.925379 18848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f32b0f087904e518a37095269dac9bc: Generated new TSK 0
I20260929 18:57:18.925588 18848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:18.928663 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18766
W20260929 18:57:18.932987 18751 connection.cc:535] client connection to 127.0.0.1:45017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45017): Connection reset by peer (error 104)
2026-09-29T18:57:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:18 dist-test-slave-q2kh krb5kdc[18865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:18 dist-test-slave-q2kh krb5kdc[18865](info): set up 2 sockets
Sep 29 18:57:18 dist-test-slave-q2kh krb5kdc[18865](info): commencing operation
krb5kdc: starting...
W20260929 18:57:19.962366 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:20 dist-test-slave-q2kh krb5kdc[18865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:20.084914 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:20.169422 18878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:20.169633 18878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:20.169657 18878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:20.170971 18878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:20.171015 18878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:20.171029 18878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:20.171043 18878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:20.171056 18878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:20.172574 18878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:20.172768 18878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:20.172936 18878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:20.177678 18883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:20.177667 18886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:20.177667 18884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:20.178017 18878 server_base.cc:1034] running on GCE node
I20260929 18:57:20.178305 18878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:20.178551 18878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:20.179744 18878 hybrid_clock.cc:648] HybridClock initialized: now 1790708240179720 us; error 36 us; skew 500 ppm
Sep 29 18:57:20 dist-test-slave-q2kh krb5kdc[18865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:20.182225 18878 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:20.182766 18878 webserver.cc:460] Webserver started at http://127.0.0.1:35993/ using document root <none> and password file <none>
I20260929 18:57:20.183024 18878 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:20.183097 18878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:20.183259 18878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:20.184350 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "0eb65f11d01746c890ad92f94693c1f5"
format_stamp: "Formatted at 2026-09-29 18:57:20 on dist-test-slave-q2kh"
server_key: "558998c5a6409c8bda61e6058fa5c095"
server_key_iv: "de2adc5e89c44c4ae244165fb9066a4c"
server_key_version: "encryptionkey@0"
I20260929 18:57:20.184743 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "0eb65f11d01746c890ad92f94693c1f5"
format_stamp: "Formatted at 2026-09-29 18:57:20 on dist-test-slave-q2kh"
server_key: "558998c5a6409c8bda61e6058fa5c095"
server_key_iv: "de2adc5e89c44c4ae244165fb9066a4c"
server_key_version: "encryptionkey@0"
I20260929 18:57:20.186146 18878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:20.187016 18893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:20.187209 18878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 18:57:20.187327 18878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "0eb65f11d01746c890ad92f94693c1f5"
format_stamp: "Formatted at 2026-09-29 18:57:20 on dist-test-slave-q2kh"
server_key: "558998c5a6409c8bda61e6058fa5c095"
server_key_iv: "de2adc5e89c44c4ae244165fb9066a4c"
server_key_version: "encryptionkey@0"
I20260929 18:57:20.187405 18878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:20.199565 18878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:20.201467 18878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:20.201644 18878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:20.205962 18878 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34047
I20260929 18:57:20.205994 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34047 every 8 connection(s)
I20260929 18:57:20.206429 18878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:20.207281 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:20.209470 18946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5: Bootstrap starting.
I20260929 18:57:20.209682 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18878
I20260929 18:57:20.209769 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:20.209913 29982 external_mini_cluster.cc:1428] Setting key 7fa3b2ef8c6ab6a1f04bcc2fa58feabf
I20260929 18:57:20.209983 18946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:20.210306 18946 log.cc:824] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:20.211122 18946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5: No bootstrap required, opened a new log
I20260929 18:57:20.213552 18946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb65f11d01746c890ad92f94693c1f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260929 18:57:20.213683 18946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:20.213706 18946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb65f11d01746c890ad92f94693c1f5, State: Initialized, Role: FOLLOWER
I20260929 18:57:20.213837 18946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb65f11d01746c890ad92f94693c1f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260929 18:57:20.213922 18946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:20.213943 18946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:20.214008 18946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:20.214740 18946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb65f11d01746c890ad92f94693c1f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260929 18:57:20.214859 18946 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb65f11d01746c890ad92f94693c1f5; no voters:
I20260929 18:57:20.215013 18946 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:20.215137 18951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:20.215273 18946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:20.215339 18951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb65f11d01746c890ad92f94693c1f5, State: Running, Role: LEADER
I20260929 18:57:20.215471 18951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb65f11d01746c890ad92f94693c1f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260929 18:57:20.216140 18952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb65f11d01746c890ad92f94693c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb65f11d01746c890ad92f94693c1f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260929 18:57:20.216248 18952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:20.216141 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb65f11d01746c890ad92f94693c1f5. Latest consensus state: current_term: 1 leader_uuid: "0eb65f11d01746c890ad92f94693c1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb65f11d01746c890ad92f94693c1f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260929 18:57:20.216485 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:20.216477 18960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:20.217162 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:20.210497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:37045 (local address 127.0.0.1:34047)
0929 18:57:20.210701 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:20.210709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:20.210739 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:57:20.211333 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:20.211342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:20.211415 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:20.211563 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:20.211829 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:20.213278 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:20.214405 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:20.215054 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:20.215174 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:20.216261 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:20.216265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:20.216266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:20.216512 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:20.216554 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:20.216570 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:20.216895 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:20.216939 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 18:57:20.216966 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260929 18:57:20.217316 18960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:20.218096 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:20.218973 18960 catalog_manager.cc:1347] Generated new cluster ID: 5f5538cf8c9745589c006619f01fa935
I20260929 18:57:20.219030 18960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:20.225976 18960 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:20.226568 18960 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:20.229400 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:20.219709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:45257 (local address 127.0.0.1:34047)
0929 18:57:20.219745 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:20.219747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:20.225706 (+ 5959us) server_negotiation.cc:407] Connection header received
0929 18:57:20.225816 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:20.225817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:20.225830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:20.225904 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:20.226179 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:20.226948 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:20.227995 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:20.228327 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:20.228412 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:20.229139 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:20.229140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:20.229140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:20.229145 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:20.229162 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:20.229167 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:20.229233 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:20.229304 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:57:20.229315 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:57:20.239878 18960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb65f11d01746c890ad92f94693c1f5: Generated new TSK 0
I20260929 18:57:20.240094 18960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:20.244511 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18878
W20260929 18:57:20.248934 18863 connection.cc:535] client connection to 127.0.0.1:34047 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34047): Connection reset by peer (error 104)
2026-09-29T18:57:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:20 dist-test-slave-q2kh krb5kdc[18977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:20 dist-test-slave-q2kh krb5kdc[18977](info): set up 2 sockets
Sep 29 18:57:20 dist-test-slave-q2kh krb5kdc[18977](info): commencing operation
krb5kdc: starting...
W20260929 18:57:21.278429 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:21 dist-test-slave-q2kh krb5kdc[18977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:57:21.399013 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:21.496177 18990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:21.496420 18990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:21.496445 18990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:21.497555 18990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:21.497591 18990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:21.497605 18990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:21.497618 18990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:21.497630 18990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:21.499171 18990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:21.499377 18990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:21.499548 18990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:21.504462 18998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:21.504490 18996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:21.504592 18990 server_base.cc:1034] running on GCE node
W20260929 18:57:21.504520 18995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:21.505038 18990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:21.505368 18990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:21.506521 18990 hybrid_clock.cc:648] HybridClock initialized: now 1790708241506503 us; error 40 us; skew 500 ppm
Sep 29 18:57:21 dist-test-slave-q2kh krb5kdc[18977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:21.509222 18990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:57:21.509723 18990 webserver.cc:460] Webserver started at http://127.0.0.1:39201/ using document root <none> and password file <none>
I20260929 18:57:21.509985 18990 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:21.510035 18990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:21.510217 18990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:21.511382 18990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "3fd0871b1f9b431a814459f049e6d354"
format_stamp: "Formatted at 2026-09-29 18:57:21 on dist-test-slave-q2kh"
server_key: "d14450fcdea6379a4330b9073bb275fd"
server_key_iv: "bf356296c43fbe68028dc1524215005c"
server_key_version: "encryptionkey@0"
I20260929 18:57:21.511765 18990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "3fd0871b1f9b431a814459f049e6d354"
format_stamp: "Formatted at 2026-09-29 18:57:21 on dist-test-slave-q2kh"
server_key: "d14450fcdea6379a4330b9073bb275fd"
server_key_iv: "bf356296c43fbe68028dc1524215005c"
server_key_version: "encryptionkey@0"
I20260929 18:57:21.513239 18990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:57:21.514112 19005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:21.514343 18990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:21.514439 18990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "3fd0871b1f9b431a814459f049e6d354"
format_stamp: "Formatted at 2026-09-29 18:57:21 on dist-test-slave-q2kh"
server_key: "d14450fcdea6379a4330b9073bb275fd"
server_key_iv: "bf356296c43fbe68028dc1524215005c"
server_key_version: "encryptionkey@0"
I20260929 18:57:21.514518 18990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:21.518118 18990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:21.520248 18990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:21.520398 18990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:21.524806 18990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37093
I20260929 18:57:21.524880 19057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37093 every 8 connection(s)
I20260929 18:57:21.525205 18990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:21.526289 19058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:21.528548 19058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354: Bootstrap starting.
I20260929 18:57:21.529078 19058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:21.529338 19058 log.cc:824] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:21.530067 19058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354: No bootstrap required, opened a new log
I20260929 18:57:21.531665 19058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd0871b1f9b431a814459f049e6d354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260929 18:57:21.531810 19058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:21.531834 19058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd0871b1f9b431a814459f049e6d354, State: Initialized, Role: FOLLOWER
I20260929 18:57:21.531991 19058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd0871b1f9b431a814459f049e6d354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260929 18:57:21.532053 19058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:21.532075 19058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:21.532130 19058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:21.532734 19058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd0871b1f9b431a814459f049e6d354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260929 18:57:21.532822 19058 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd0871b1f9b431a814459f049e6d354; no voters:
I20260929 18:57:21.533025 19058 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:21.533125 19061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:21.533295 19061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [term 1 LEADER]: Becoming Leader. State: Replica: 3fd0871b1f9b431a814459f049e6d354, State: Running, Role: LEADER
I20260929 18:57:21.533351 19058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:21.533397 19061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd0871b1f9b431a814459f049e6d354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } }
I20260929 18:57:21.533790 19062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fd0871b1f9b431a814459f049e6d354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd0871b1f9b431a814459f049e6d354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } } }
I20260929 18:57:21.533890 19062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:21.533934 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 18990
I20260929 18:57:21.534052 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:21.534062 19063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fd0871b1f9b431a814459f049e6d354. Latest consensus state: current_term: 1 leader_uuid: "3fd0871b1f9b431a814459f049e6d354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd0871b1f9b431a814459f049e6d354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37093 } } }
I20260929 18:57:21.534119 19063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:21.534279 29982 external_mini_cluster.cc:1428] Setting key fb6e7ad6f48c1db0691a932d11985fd7
W20260929 18:57:21.536576 19079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 18:57:21.536585 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:21.534845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:56915 (local address 127.0.0.1:37093)
0929 18:57:21.534997 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:21.535006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:21.535049 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:57:21.535173 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:21.535179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:21.535204 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:21.535317 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:21.536031 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:21.536034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:21.536034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:21.536037 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:21.536071 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:21.536082 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:21.536154 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:21.536318 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 18:57:21.536364 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":80,"threads_started":1}
W20260929 18:57:21.536823 19079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:21.536883 19065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:21.537050 19065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:21.537797 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:21.538633 19065 catalog_manager.cc:1347] Generated new cluster ID: 8c1c7e1bf61b4bfb8ea1e7f026574eab
I20260929 18:57:21.538681 19065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:21.540607 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:21.539541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:52211 (local address 127.0.0.1:37093)
0929 18:57:21.539578 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:21.539580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:21.539645 (+ 65us) server_negotiation.cc:407] Connection header received
0929 18:57:21.539699 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:21.539700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:21.539704 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:21.539776 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:21.540386 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:21.540387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:21.540387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:21.540391 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:21.540407 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:21.540411 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:21.540468 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:21.540530 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 18:57:21.540544 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:21.570361 19065 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:21.570592 19065 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:21.581370 19065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fd0871b1f9b431a814459f049e6d354: Generated new TSK 0
I20260929 18:57:21.581611 19065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:21.589352 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 18990
2026-09-29T18:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:21 dist-test-slave-q2kh krb5kdc[19089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:21 dist-test-slave-q2kh krb5kdc[19089](info): set up 2 sockets
Sep 29 18:57:21 dist-test-slave-q2kh krb5kdc[19089](info): commencing operation
krb5kdc: starting...
W20260929 18:57:22.622375 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:22 dist-test-slave-q2kh krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:22.749498 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:40317
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:39593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:22.830370 19102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:22.830603 19102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:22.830641 19102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:22.831820 19102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:22.831913 19102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:22.831940 19102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:22.831970 19102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:22.831995 19102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:22.833470 19102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:39593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:40317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:22.833748 19102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:22.833977 19102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:22.838838 19110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:22.838855 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:22.839021 19102 server_base.cc:1034] running on GCE node
W20260929 18:57:22.839109 19108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:22.839339 19102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:22.839578 19102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:22.840758 19102 hybrid_clock.cc:648] HybridClock initialized: now 1790708242840758 us; error 60 us; skew 500 ppm
Sep 29 18:57:22 dist-test-slave-q2kh krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708242, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:22.843621 19102 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:22.844100 19102 webserver.cc:460] Webserver started at http://127.29.71.190:33033/ using document root <none> and password file <none>
I20260929 18:57:22.844296 19102 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:22.844340 19102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:22.844432 19102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:22.845515 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "73c6558772ef4ecf85311eae6129db25"
format_stamp: "Formatted at 2026-09-29 18:57:22 on dist-test-slave-q2kh"
server_key: "3774dd17c8f72d20c23d98777a466682"
server_key_iv: "2e1d1d28ecc6a43788b3b511a834f49e"
server_key_version: "encryptionkey@0"
I20260929 18:57:22.845839 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "73c6558772ef4ecf85311eae6129db25"
format_stamp: "Formatted at 2026-09-29 18:57:22 on dist-test-slave-q2kh"
server_key: "3774dd17c8f72d20c23d98777a466682"
server_key_iv: "2e1d1d28ecc6a43788b3b511a834f49e"
server_key_version: "encryptionkey@0"
I20260929 18:57:22.847332 19102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:22.848104 19117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:22.848345 19102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:22.848451 19102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "73c6558772ef4ecf85311eae6129db25"
format_stamp: "Formatted at 2026-09-29 18:57:22 on dist-test-slave-q2kh"
server_key: "3774dd17c8f72d20c23d98777a466682"
server_key_iv: "2e1d1d28ecc6a43788b3b511a834f49e"
server_key_version: "encryptionkey@0"
I20260929 18:57:22.848529 19102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:22.860188 19102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:22.862128 19102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:22.862360 19102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:22.866376 19102 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:40317
I20260929 18:57:22.866433 19169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:40317 every 8 connection(s)
I20260929 18:57:22.866776 19102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:22.867622 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:22.869998 19170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25: Bootstrap starting.
I20260929 18:57:22.870643 19170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:22.870949 19170 log.cc:824] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:22.871661 19170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25: No bootstrap required, opened a new log
I20260929 18:57:22.873521 19170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c6558772ef4ecf85311eae6129db25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40317 } }
I20260929 18:57:22.873682 19170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:22.873706 19170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c6558772ef4ecf85311eae6129db25, State: Initialized, Role: FOLLOWER
I20260929 18:57:22.873865 19170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c6558772ef4ecf85311eae6129db25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40317 } }
I20260929 18:57:22.873944 19170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:22.873966 19170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:22.874042 19170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:22.874516 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19102
I20260929 18:57:22.874615 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:22.874818 29982 external_mini_cluster.cc:1428] Setting key 1d5ef73de2dd070ae817b25d506c4ca8
I20260929 18:57:22.875178 19170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c6558772ef4ecf85311eae6129db25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40317 } }
I20260929 18:57:22.875339 19170 leader_election.cc:302] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73c6558772ef4ecf85311eae6129db25; no voters:
I20260929 18:57:22.875512 19170 leader_election.cc:288] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:22.875679 19175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:22.875792 19170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:22.875852 19175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [term 1 LEADER]: Becoming Leader. State: Replica: 73c6558772ef4ecf85311eae6129db25, State: Running, Role: LEADER
I20260929 18:57:22.876097 19175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c6558772ef4ecf85311eae6129db25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40317 } }
I20260929 18:57:22.876566 19177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73c6558772ef4ecf85311eae6129db25. Latest consensus state: current_term: 1 leader_uuid: "73c6558772ef4ecf85311eae6129db25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c6558772ef4ecf85311eae6129db25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40317 } } }
I20260929 18:57:22.876681 19177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:22.876565 19176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73c6558772ef4ecf85311eae6129db25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c6558772ef4ecf85311eae6129db25" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40317 } } }
I20260929 18:57:22.876827 19176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:22.876991 19184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:22.877846 19184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:22.879566 19184 catalog_manager.cc:1347] Generated new cluster ID: 19cc0a7cdc454079bec4ea64dba663d9
I20260929 18:57:22.879624 19184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 18:57:22 dist-test-slave-q2kh krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:22.887040 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:22.875356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37209 (local address 127.29.71.190:40317)
0929 18:57:22.875558 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:22.875568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:22.875615 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:57:22.875755 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:22.875763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:22.875852 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:22.876983 (+ 1131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:22.877134 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:22.878902 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:22.879938 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:22.880410 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:22.880562 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:22.883230 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:22.883234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:22.883235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:22.883246 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:22.885142 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:22.885596 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:22.885599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:22.885599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:22.885709 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:22.885973 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:22.885974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:22.885974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:22.886316 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:22.886508 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:22.886669 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:57:22.886789 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260929 18:57:22.888219 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:22.896350 19184 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:22.897171 19184 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:22.897269 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:22.889707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46427 (local address 127.29.71.190:40317)
0929 18:57:22.889747 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:22.889749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:22.890361 (+ 612us) server_negotiation.cc:407] Connection header received
0929 18:57:22.890442 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:22.890443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:22.890453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:22.890515 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:22.891078 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:22.891617 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:22.892410 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:22.892668 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:22.892769 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:22.894070 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:22.894071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:22.894071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:22.894080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:22.895730 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:22.896230 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:22.896231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:22.896231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:22.896318 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:22.896586 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:22.896587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:22.896588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:22.896716 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:22.896853 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:22.896994 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 18:57:22.897086 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":20}
I20260929 18:57:22.905932 19184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73c6558772ef4ecf85311eae6129db25: Generated new TSK 0
I20260929 18:57:22.906144 19184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:22.911382 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19102
W20260929 18:57:22.916100 19087 connection.cc:535] client connection to 127.29.71.190:40317 recv error: Network error: failed to read from TLS socket (remote: 127.29.71.190:40317): Connection reset by peer (error 104)
2026-09-29T18:57:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:22 dist-test-slave-q2kh krb5kdc[19201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:22 dist-test-slave-q2kh krb5kdc[19201](info): set up 2 sockets
Sep 29 18:57:22 dist-test-slave-q2kh krb5kdc[19201](info): commencing operation
krb5kdc: starting...
W20260929 18:57:24.042596 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:24 dist-test-slave-q2kh krb5kdc[19201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:24.163363 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41545
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:41545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:24.243083 19214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:24.243340 19214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:24.243378 19214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:24.244520 19214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:24.244601 19214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:24.244626 19214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:24.244649 19214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:24.244707 19214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:24.246199 19214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:41545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:24.246490 19214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:24.246747 19214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:24.251734 19222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:24.251741 19220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:24.251770 19219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:24.252012 19214 server_base.cc:1034] running on GCE node
I20260929 18:57:24.252300 19214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:24.252604 19214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:24.253772 19214 hybrid_clock.cc:648] HybridClock initialized: now 1790708244253750 us; error 36 us; skew 500 ppm
Sep 29 18:57:24 dist-test-slave-q2kh krb5kdc[19201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708244, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:24.260196 19214 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:24.260713 19214 webserver.cc:460] Webserver started at http://127.29.71.190:37597/ using document root <none> and password file <none>
I20260929 18:57:24.260967 19214 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:24.261039 19214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:24.261200 19214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:24.262391 19214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "176b61e7b56e4d769cd461cdc417f4b1"
format_stamp: "Formatted at 2026-09-29 18:57:24 on dist-test-slave-q2kh"
server_key: "b31c57dbb42fb4e1a4ba8da8b3e5cd75"
server_key_iv: "6e828db531a464c7d8cc7c6cf1d33bfd"
server_key_version: "encryptionkey@0"
I20260929 18:57:24.262753 19214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "176b61e7b56e4d769cd461cdc417f4b1"
format_stamp: "Formatted at 2026-09-29 18:57:24 on dist-test-slave-q2kh"
server_key: "b31c57dbb42fb4e1a4ba8da8b3e5cd75"
server_key_iv: "6e828db531a464c7d8cc7c6cf1d33bfd"
server_key_version: "encryptionkey@0"
I20260929 18:57:24.264150 19214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:57:24.265038 19229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:24.265242 19214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:24.265367 19214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "176b61e7b56e4d769cd461cdc417f4b1"
format_stamp: "Formatted at 2026-09-29 18:57:24 on dist-test-slave-q2kh"
server_key: "b31c57dbb42fb4e1a4ba8da8b3e5cd75"
server_key_iv: "6e828db531a464c7d8cc7c6cf1d33bfd"
server_key_version: "encryptionkey@0"
I20260929 18:57:24.265456 19214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:24.275249 19214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:24.277791 19214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:24.277961 19214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:24.281893 19214 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:41545
I20260929 18:57:24.281926 19281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:41545 every 8 connection(s)
I20260929 18:57:24.282356 19214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:24.283253 19282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:24.285493 19282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1: Bootstrap starting.
I20260929 18:57:24.286079 19282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:24.286396 19282 log.cc:824] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:24.287138 19282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1: No bootstrap required, opened a new log
I20260929 18:57:24.287851 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19214
I20260929 18:57:24.287958 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:24.288138 29982 external_mini_cluster.cc:1428] Setting key 99367df19e059ecb8e90a78299cfe75f
I20260929 18:57:24.289268 19282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176b61e7b56e4d769cd461cdc417f4b1" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41545 } }
I20260929 18:57:24.289433 19282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:24.289495 19282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176b61e7b56e4d769cd461cdc417f4b1, State: Initialized, Role: FOLLOWER
I20260929 18:57:24.289651 19282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176b61e7b56e4d769cd461cdc417f4b1" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41545 } }
I20260929 18:57:24.289790 19282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:24.289841 19282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:24.289915 19282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:24.290581 19282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176b61e7b56e4d769cd461cdc417f4b1" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41545 } }
I20260929 18:57:24.290712 19282 leader_election.cc:302] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 176b61e7b56e4d769cd461cdc417f4b1; no voters:
I20260929 18:57:24.290896 19282 leader_election.cc:288] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:24.290927 19287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:24.291116 19287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [term 1 LEADER]: Becoming Leader. State: Replica: 176b61e7b56e4d769cd461cdc417f4b1, State: Running, Role: LEADER
I20260929 18:57:24.291146 19282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:24.291281 19287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176b61e7b56e4d769cd461cdc417f4b1" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41545 } }
I20260929 18:57:24.292654 19289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 176b61e7b56e4d769cd461cdc417f4b1. Latest consensus state: current_term: 1 leader_uuid: "176b61e7b56e4d769cd461cdc417f4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176b61e7b56e4d769cd461cdc417f4b1" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41545 } } }
I20260929 18:57:24.292661 19288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "176b61e7b56e4d769cd461cdc417f4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "176b61e7b56e4d769cd461cdc417f4b1" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41545 } } }
I20260929 18:57:24.293002 19289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:24.293009 19288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1 [sys.catalog]: This master's current role is: LEADER
W20260929 18:57:24.293259 19302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:24.293318 19302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:24.293421 19304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:24.293639 19304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:24.294772 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:24.288692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51389 (local address 127.29.71.190:41545)
0929 18:57:24.288892 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:24.288909 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:24.288933 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:57:24.289256 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:24.289263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:24.289307 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:24.289446 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:24.289637 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:24.291171 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:24.292407 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:24.293047 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:24.293158 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:24.293984 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:24.293989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:24.293989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:24.294236 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:24.294275 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:24.294291 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:24.294378 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:24.294535 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 18:57:24.294563 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260929 18:57:24.295626 19304 catalog_manager.cc:1347] Generated new cluster ID: cc71f5005ff44599b1ab36371630c1ea
I20260929 18:57:24.295680 19304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:24.295851 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:24.301921 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:24.297308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40539 (local address 127.29.71.190:41545)
0929 18:57:24.297339 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:24.297341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:24.298336 (+ 995us) server_negotiation.cc:407] Connection header received
0929 18:57:24.298486 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:24.298487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:24.298500 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:24.298581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:24.298733 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:24.299528 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:24.300529 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:24.300862 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:24.300938 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:24.301654 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:24.301655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:24.301655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:24.301660 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:24.301676 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:24.301681 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:24.301754 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:24.301829 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 18:57:24.301841 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 18:57:24.311506 19304 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:24.312244 19304 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:24.322671 19304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 176b61e7b56e4d769cd461cdc417f4b1: Generated new TSK 0
I20260929 18:57:24.322925 19304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:24.330101 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19214
W20260929 18:57:24.334340 19199 connection.cc:535] client connection to 127.29.71.190:41545 recv error: Network error: failed to read from TLS socket (remote: 127.29.71.190:41545): Connection reset by peer (error 104)
2026-09-29T18:57:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:24 dist-test-slave-q2kh krb5kdc[19313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:24 dist-test-slave-q2kh krb5kdc[19313](info): set up 2 sockets
Sep 29 18:57:24 dist-test-slave-q2kh krb5kdc[19313](info): commencing operation
krb5kdc: starting...
W20260929 18:57:25.374285 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:25 dist-test-slave-q2kh krb5kdc[19313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:25.493108 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41197
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:41197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:25.573765 19326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:25.574004 19326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:25.574043 19326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:25.575214 19326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:25.575277 19326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:25.575301 19326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:25.575335 19326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:25.575362 19326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:25.576804 19326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:41197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:41197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:25.577069 19326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:25.577292 19326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:57:25.582250 19326 server_base.cc:1034] running on GCE node
W20260929 18:57:25.582288 19332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:25.582221 19331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:25.582237 19334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:25.582665 19326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:25.582998 19326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:25.584156 19326 hybrid_clock.cc:648] HybridClock initialized: now 1790708245584139 us; error 33 us; skew 500 ppm
Sep 29 18:57:25 dist-test-slave-q2kh krb5kdc[19313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708245, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:25.586589 19326 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:25.587039 19326 webserver.cc:460] Webserver started at http://127.29.71.190:46527/ using document root <none> and password file <none>
I20260929 18:57:25.587301 19326 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:25.587371 19326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:25.587517 19326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:25.588558 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "8c4902eea93745469561cf3685b94b60"
format_stamp: "Formatted at 2026-09-29 18:57:25 on dist-test-slave-q2kh"
server_key: "73bf74f693c23675378659f32b58ad2c"
server_key_iv: "ba4b7941d9936c191eded7e5c8d43258"
server_key_version: "encryptionkey@0"
I20260929 18:57:25.588948 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "8c4902eea93745469561cf3685b94b60"
format_stamp: "Formatted at 2026-09-29 18:57:25 on dist-test-slave-q2kh"
server_key: "73bf74f693c23675378659f32b58ad2c"
server_key_iv: "ba4b7941d9936c191eded7e5c8d43258"
server_key_version: "encryptionkey@0"
I20260929 18:57:25.590410 19326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:25.591207 19341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:25.591396 19326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:25.591493 19326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "8c4902eea93745469561cf3685b94b60"
format_stamp: "Formatted at 2026-09-29 18:57:25 on dist-test-slave-q2kh"
server_key: "73bf74f693c23675378659f32b58ad2c"
server_key_iv: "ba4b7941d9936c191eded7e5c8d43258"
server_key_version: "encryptionkey@0"
I20260929 18:57:25.591576 19326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:25.594857 19326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:25.596735 19326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:25.596896 19326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:25.600898 19326 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:41197
I20260929 18:57:25.600929 19393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:41197 every 8 connection(s)
I20260929 18:57:25.601254 19326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:25.602397 19394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:25.604688 19394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60: Bootstrap starting.
I20260929 18:57:25.605237 19394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:25.605476 19394 log.cc:824] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:25.606123 19394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60: No bootstrap required, opened a new log
I20260929 18:57:25.607630 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19326
I20260929 18:57:25.607760 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:25.607939 29982 external_mini_cluster.cc:1428] Setting key 59955edcb9e81c5f1dac73d901728706
I20260929 18:57:25.607980 19394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4902eea93745469561cf3685b94b60" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41197 } }
I20260929 18:57:25.608184 19394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:25.608242 19394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4902eea93745469561cf3685b94b60, State: Initialized, Role: FOLLOWER
I20260929 18:57:25.608408 19394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4902eea93745469561cf3685b94b60" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41197 } }
I20260929 18:57:25.608546 19394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:25.608603 19394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:25.608676 19394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:25.609856 19394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4902eea93745469561cf3685b94b60" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41197 } }
I20260929 18:57:25.609972 19394 leader_election.cc:302] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4902eea93745469561cf3685b94b60; no voters:
I20260929 18:57:25.610129 19394 leader_election.cc:288] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:25.610193 19399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:25.610229 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:25.608483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57445 (local address 127.29.71.190:41197)
0929 18:57:25.608697 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:25.608706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:25.608740 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:57:25.608867 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:25.608875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:25.608905 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:25.609066 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:25.609682 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:25.609686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:25.609686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:25.609692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:25.609739 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:25.609755 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:25.609840 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:25.609916 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:57:25.609959 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260929 18:57:25.610333 19399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [term 1 LEADER]: Becoming Leader. State: Replica: 8c4902eea93745469561cf3685b94b60, State: Running, Role: LEADER
I20260929 18:57:25.610399 19394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:25.610499 19399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4902eea93745469561cf3685b94b60" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41197 } }
I20260929 18:57:25.611037 19399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4902eea93745469561cf3685b94b60. Latest consensus state: current_term: 1 leader_uuid: "8c4902eea93745469561cf3685b94b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4902eea93745469561cf3685b94b60" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41197 } } }
I20260929 18:57:25.611057 19401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c4902eea93745469561cf3685b94b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4902eea93745469561cf3685b94b60" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 41197 } } }
I20260929 18:57:25.611227 19401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:25.611245 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:57:25.612363 19415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 18:57:25.611163 19399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60 [sys.catalog]: This master's current role is: LEADER
W20260929 18:57:25.612641 19415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:25.612779 19408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:25.612953 19408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:25.614439 19408 catalog_manager.cc:1347] Generated new cluster ID: 5b35044a77ce486ba86469ad1a1bc822
I20260929 18:57:25.614495 19408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:25.614691 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:25.612829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48297 (local address 127.29.71.190:41197)
0929 18:57:25.612868 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:25.612870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:25.613712 (+ 842us) server_negotiation.cc:407] Connection header received
0929 18:57:25.613776 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:25.613777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:25.613782 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:25.613844 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:25.614470 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:25.614471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:25.614472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:25.614476 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:25.614489 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:25.614494 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:25.614541 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:25.614621 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 18:57:25.614634 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:57:25.627282 19408 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:25.627499 19408 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:25.635836 19408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c4902eea93745469561cf3685b94b60: Generated new TSK 0
I20260929 18:57:25.636030 19408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:25.640038 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19326
2026-09-29T18:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:25 dist-test-slave-q2kh krb5kdc[19425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:25 dist-test-slave-q2kh krb5kdc[19425](info): set up 2 sockets
Sep 29 18:57:25 dist-test-slave-q2kh krb5kdc[19425](info): commencing operation
krb5kdc: starting...
W20260929 18:57:26.690475 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:26 dist-test-slave-q2kh krb5kdc[19425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:26.809762 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:39945
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:26.889101 19438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:26.889310 19438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:26.889334 19438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:26.890421 19438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:26.890453 19438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:26.890466 19438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:26.890479 19438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:26.890491 19438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:26.891891 19438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:39945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:26.892074 19438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:26.892232 19438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:26.897352 19443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:26.897349 19446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:26.897389 19444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:26.897517 19438 server_base.cc:1034] running on GCE node
I20260929 18:57:26.897764 19438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:26.898049 19438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:26.899240 19438 hybrid_clock.cc:648] HybridClock initialized: now 1790708246899222 us; error 43 us; skew 500 ppm
Sep 29 18:57:26 dist-test-slave-q2kh krb5kdc[19425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708246, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:26.901859 19438 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:26.902383 19438 webserver.cc:460] Webserver started at http://127.29.71.190:45723/ using document root <none> and password file <none>
I20260929 18:57:26.902629 19438 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:26.902710 19438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:26.902868 19438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:26.904044 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "5af123e9168f40f79dd3231558b070d2"
format_stamp: "Formatted at 2026-09-29 18:57:26 on dist-test-slave-q2kh"
server_key: "d33943074cafef6ad459579c880d45ba"
server_key_iv: "0a701991a327e4bb5dde456da094969c"
server_key_version: "encryptionkey@0"
I20260929 18:57:26.904429 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "5af123e9168f40f79dd3231558b070d2"
format_stamp: "Formatted at 2026-09-29 18:57:26 on dist-test-slave-q2kh"
server_key: "d33943074cafef6ad459579c880d45ba"
server_key_iv: "0a701991a327e4bb5dde456da094969c"
server_key_version: "encryptionkey@0"
I20260929 18:57:26.905807 19438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:26.906630 19453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:26.906816 19438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:26.906908 19438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "5af123e9168f40f79dd3231558b070d2"
format_stamp: "Formatted at 2026-09-29 18:57:26 on dist-test-slave-q2kh"
server_key: "d33943074cafef6ad459579c880d45ba"
server_key_iv: "0a701991a327e4bb5dde456da094969c"
server_key_version: "encryptionkey@0"
I20260929 18:57:26.906993 19438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:26.914913 19438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:26.916816 19438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:26.916977 19438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:26.920753 19438 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:39945
I20260929 18:57:26.920815 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:39945 every 8 connection(s)
I20260929 18:57:26.921175 19438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:26.922049 19506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:26.924055 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19438
I20260929 18:57:26.924185 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:26.924381 29982 external_mini_cluster.cc:1428] Setting key f913692d6685c540fe737db6a2276f90
I20260929 18:57:26.924371 19506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2: Bootstrap starting.
I20260929 18:57:26.925134 19506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:26.925479 19506 log.cc:824] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:26.926301 19506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2: No bootstrap required, opened a new log
I20260929 18:57:26.928774 19506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af123e9168f40f79dd3231558b070d2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39945 } }
I20260929 18:57:26.928911 19506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:26.928936 19506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af123e9168f40f79dd3231558b070d2, State: Initialized, Role: FOLLOWER
I20260929 18:57:26.929081 19506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af123e9168f40f79dd3231558b070d2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39945 } }
I20260929 18:57:26.929163 19506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:26.929209 19506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:26.929286 19506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:26.930006 19506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af123e9168f40f79dd3231558b070d2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39945 } }
I20260929 18:57:26.930119 19506 leader_election.cc:302] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af123e9168f40f79dd3231558b070d2; no voters:
I20260929 18:57:26.930352 19506 leader_election.cc:288] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:26.930380 19511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:26.930660 19506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:26.930704 19511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5af123e9168f40f79dd3231558b070d2, State: Running, Role: LEADER
I20260929 18:57:26.930788 19511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af123e9168f40f79dd3231558b070d2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39945 } }
I20260929 18:57:26.931878 19513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af123e9168f40f79dd3231558b070d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af123e9168f40f79dd3231558b070d2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39945 } } }
I20260929 18:57:26.932307 19513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:26.932224 19512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af123e9168f40f79dd3231558b070d2. Latest consensus state: current_term: 1 leader_uuid: "5af123e9168f40f79dd3231558b070d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af123e9168f40f79dd3231558b070d2" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 39945 } } }
Sep 29 18:57:26 dist-test-slave-q2kh krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
W20260929 18:57:26.932723 19526 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:26.933161 19526 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:26.932934 19512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:26.933244 19527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:26.933427 19527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:26.935094 19527 catalog_manager.cc:1347] Generated new cluster ID: 039ee5674a5c4450aa6182f97ea37823
I20260929 18:57:26.935148 19527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:26.936681 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:26.924941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39135 (local address 127.29.71.190:39945)
0929 18:57:26.925153 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:26.925163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:26.925197 (+ 34us) server_negotiation.cc:407] Connection header received
0929 18:57:26.925948 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:26.925957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:26.926023 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:26.926218 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:26.926869 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:26.928370 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:26.929334 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:26.929949 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:26.930073 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:26.932924 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:26.932928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:26.932928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:26.932940 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:26.935024 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:26.935428 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:26.935431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:26.935432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:26.935529 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:26.935798 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:26.935799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:26.935799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:26.936052 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:26.936245 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:26.936316 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:57:26.936465 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260929 18:57:26.938805 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:26.945712 19527 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:26.946280 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:26.940262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53851 (local address 127.29.71.190:39945)
0929 18:57:26.940298 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:26.940300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:26.940308 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:57:26.940329 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:26.940330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:26.940339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:26.940382 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:26.940893 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:26.941368 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:26.942068 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:26.942383 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:26.942468 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:26.943621 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:26.943622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:26.943623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:26.943629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:26.945006 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:26.945397 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:26.945398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:26.945398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:26.945461 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:26.945712 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:26.945712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:26.945712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:26.945863 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:26.945958 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:26.946090 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 18:57:26.946194 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:26.946439 19527 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:26.949515 19527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af123e9168f40f79dd3231558b070d2: Generated new TSK 0
I20260929 18:57:26.949688 19527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:26.951907 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19438
2026-09-29T18:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:27 dist-test-slave-q2kh krb5kdc[19537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:27 dist-test-slave-q2kh krb5kdc[19537](info): set up 2 sockets
Sep 29 18:57:27 dist-test-slave-q2kh krb5kdc[19537](info): commencing operation
krb5kdc: starting...
W20260929 18:57:27.974797 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:28 dist-test-slave-q2kh krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:28.097595 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:36889
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:40097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:36889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:28.186460 19550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:28.186678 19550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:28.186715 19550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:28.187867 19550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:28.187934 19550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:28.187959 19550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:28.187992 19550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:28.188021 19550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:28.189451 19550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:40097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:36889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:28.189726 19550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:28.189963 19550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:28.195084 19556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:28.195093 19555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:28.195156 19550 server_base.cc:1034] running on GCE node
W20260929 18:57:28.195106 19558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:28.195519 19550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:28.195813 19550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:28.196985 19550 hybrid_clock.cc:648] HybridClock initialized: now 1790708248196980 us; error 36 us; skew 500 ppm
Sep 29 18:57:28 dist-test-slave-q2kh krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708248, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:28.199576 19550 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:28.200040 19550 webserver.cc:460] Webserver started at http://127.29.71.190:41859/ using document root <none> and password file <none>
I20260929 18:57:28.200268 19550 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:28.200315 19550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:28.200445 19550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:28.201507 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "6bda7ae702b04eb38be6deccbee7145d"
format_stamp: "Formatted at 2026-09-29 18:57:28 on dist-test-slave-q2kh"
server_key: "880eb6726cfcd11d2ec194dec9161dfc"
server_key_iv: "d4f62bd0fd88f5a4aeba1c9b99faa11e"
server_key_version: "encryptionkey@0"
I20260929 18:57:28.201867 19550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "6bda7ae702b04eb38be6deccbee7145d"
format_stamp: "Formatted at 2026-09-29 18:57:28 on dist-test-slave-q2kh"
server_key: "880eb6726cfcd11d2ec194dec9161dfc"
server_key_iv: "d4f62bd0fd88f5a4aeba1c9b99faa11e"
server_key_version: "encryptionkey@0"
I20260929 18:57:28.203334 19550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:57:28.204164 19565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:28.204414 19550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:28.204530 19550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "6bda7ae702b04eb38be6deccbee7145d"
format_stamp: "Formatted at 2026-09-29 18:57:28 on dist-test-slave-q2kh"
server_key: "880eb6726cfcd11d2ec194dec9161dfc"
server_key_iv: "d4f62bd0fd88f5a4aeba1c9b99faa11e"
server_key_version: "encryptionkey@0"
I20260929 18:57:28.204627 19550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:28.239086 19550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:28.241720 19550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:28.241907 19550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:28.246485 19550 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:36889
I20260929 18:57:28.246521 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:36889 every 8 connection(s)
I20260929 18:57:28.246901 19550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:28.247812 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:28.250262 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d: Bootstrap starting.
I20260929 18:57:28.250845 19618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:28.251132 19618 log.cc:824] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:28.251896 19618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d: No bootstrap required, opened a new log
I20260929 18:57:28.253134 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19550
I20260929 18:57:28.253255 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:28.253436 29982 external_mini_cluster.cc:1428] Setting key a2249c5846d6fb3704ebbef4e33c37d6
I20260929 18:57:28.253928 19618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bda7ae702b04eb38be6deccbee7145d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36889 } }
I20260929 18:57:28.254136 19618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:28.254266 19618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bda7ae702b04eb38be6deccbee7145d, State: Initialized, Role: FOLLOWER
I20260929 18:57:28.254422 19618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bda7ae702b04eb38be6deccbee7145d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36889 } }
I20260929 18:57:28.254524 19618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:28.254568 19618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:28.254626 19618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:28.255241 19618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bda7ae702b04eb38be6deccbee7145d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36889 } }
I20260929 18:57:28.255328 19618 leader_election.cc:302] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bda7ae702b04eb38be6deccbee7145d; no voters:
I20260929 18:57:28.255498 19618 leader_election.cc:288] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:28.255542 19623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:28.255698 19623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [term 1 LEADER]: Becoming Leader. State: Replica: 6bda7ae702b04eb38be6deccbee7145d, State: Running, Role: LEADER
I20260929 18:57:28.255771 19618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:28.255838 19623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bda7ae702b04eb38be6deccbee7145d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36889 } }
I20260929 18:57:28.256150 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bda7ae702b04eb38be6deccbee7145d. Latest consensus state: current_term: 1 leader_uuid: "6bda7ae702b04eb38be6deccbee7145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bda7ae702b04eb38be6deccbee7145d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36889 } } }
I20260929 18:57:28.256209 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:28.256165 19624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bda7ae702b04eb38be6deccbee7145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bda7ae702b04eb38be6deccbee7145d" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 36889 } } }
I20260929 18:57:28.256474 19624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:28.257248 19629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:28.257803 19629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:28.259634 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:28.254007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33083 (local address 127.29.71.190:36889)
0929 18:57:28.254230 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:28.254240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:28.254277 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:57:28.254428 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:28.254435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:28.254487 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:28.254637 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:28.254963 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:28.256308 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:28.257624 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:28.258072 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:28.258214 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:28.259054 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:28.259056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:28.259057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:28.259218 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:28.259249 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:28.259260 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:28.259312 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:28.259425 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 18:57:28.259475 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
I20260929 18:57:28.260601 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:28.262668 19629 catalog_manager.cc:1347] Generated new cluster ID: 838a63fa196241d585952e62c92fe648
I20260929 18:57:28.262734 19629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:28.268704 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:28.261934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43025 (local address 127.29.71.190:36889)
0929 18:57:28.261967 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:28.261969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:28.262362 (+ 393us) server_negotiation.cc:407] Connection header received
0929 18:57:28.262398 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:28.262399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:28.262411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:28.262492 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:28.266261 (+ 3769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:28.266700 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:28.267438 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:28.267632 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:28.267718 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:28.268429 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:28.268430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:28.268431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:28.268436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:28.268450 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:28.268454 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:28.268507 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:28.268571 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 18:57:28.268593 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 18:57:28.276881 19629 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:28.277622 19629 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:28.281236 19629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bda7ae702b04eb38be6deccbee7145d: Generated new TSK 0
I20260929 18:57:28.281423 19629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:28.284632 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19550
2026-09-29T18:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:28 dist-test-slave-q2kh krb5kdc[19649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:28 dist-test-slave-q2kh krb5kdc[19649](info): set up 2 sockets
Sep 29 18:57:28 dist-test-slave-q2kh krb5kdc[19649](info): commencing operation
krb5kdc: starting...
W20260929 18:57:29.366425 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:29 dist-test-slave-q2kh krb5kdc[19649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:29.485169 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34075
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:32855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:34075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:29.564937 19662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:29.565133 19662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:29.565155 19662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:29.566219 19662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:29.566252 19662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:29.566267 19662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:29.566279 19662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:29.566291 19662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:29.567623 19662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:32855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:34075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:34075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:29.567806 19662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:29.567971 19662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:29.572599 19670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:29.572593 19667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:29.572642 19668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:29.572924 19662 server_base.cc:1034] running on GCE node
I20260929 18:57:29.573168 19662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:29.573439 19662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:29.574577 19662 hybrid_clock.cc:648] HybridClock initialized: now 1790708249574565 us; error 28 us; skew 500 ppm
Sep 29 18:57:29 dist-test-slave-q2kh krb5kdc[19649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708249, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:29.581745 19662 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:29.582274 19662 webserver.cc:460] Webserver started at http://127.29.71.190:44161/ using document root <none> and password file <none>
I20260929 18:57:29.582506 19662 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:29.582571 19662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:29.582712 19662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:29.583762 19662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "f5778636371b48be9151627adb85a5b5"
format_stamp: "Formatted at 2026-09-29 18:57:29 on dist-test-slave-q2kh"
server_key: "95ef80b5d7827591d6e1c95164c6916c"
server_key_iv: "7e942188fbf83f5cb26d150aec1bd0c0"
server_key_version: "encryptionkey@0"
I20260929 18:57:29.584131 19662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "f5778636371b48be9151627adb85a5b5"
format_stamp: "Formatted at 2026-09-29 18:57:29 on dist-test-slave-q2kh"
server_key: "95ef80b5d7827591d6e1c95164c6916c"
server_key_iv: "7e942188fbf83f5cb26d150aec1bd0c0"
server_key_version: "encryptionkey@0"
I20260929 18:57:29.585451 19662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:29.586335 19677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:29.586539 19662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:29.586642 19662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "f5778636371b48be9151627adb85a5b5"
format_stamp: "Formatted at 2026-09-29 18:57:29 on dist-test-slave-q2kh"
server_key: "95ef80b5d7827591d6e1c95164c6916c"
server_key_iv: "7e942188fbf83f5cb26d150aec1bd0c0"
server_key_version: "encryptionkey@0"
I20260929 18:57:29.586730 19662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:29.590082 19662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:29.592010 19662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:29.592159 19662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:29.596025 19662 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:34075
I20260929 18:57:29.596060 19729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:34075 every 8 connection(s)
I20260929 18:57:29.596383 19662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:29.597348 19730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:29.599552 19730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5: Bootstrap starting.
I20260929 18:57:29.599812 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19662
I20260929 18:57:29.599928 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:29.600131 29982 external_mini_cluster.cc:1428] Setting key bfc5aa9ffda85fbbfccbe37b4eecbb46
I20260929 18:57:29.600234 19730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:29.600528 19730 log.cc:824] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:29.601199 19730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5: No bootstrap required, opened a new log
I20260929 18:57:29.602526 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:29.600645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58259 (local address 127.29.71.190:34075)
0929 18:57:29.600822 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:29.600830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:29.600859 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:57:29.601267 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:29.601293 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:29.601327 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:29.601481 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:29.602119 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:29.602123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:29.602123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:29.602127 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:29.602194 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:29.602208 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:29.602283 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:29.602322 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 18:57:29.602362 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260929 18:57:29.603564 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:29.603754 19730 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5778636371b48be9151627adb85a5b5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34075 } }
I20260929 18:57:29.603905 19730 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:57:29.603953 19730 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5778636371b48be9151627adb85a5b5, State: Initialized, Role: FOLLOWER
I20260929 18:57:29.604063 19730 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5778636371b48be9151627adb85a5b5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34075 } }
I20260929 18:57:29.604141 19730 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:57:29.604182 19730 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:57:29.604229 19730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:29.604738 19730 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5778636371b48be9151627adb85a5b5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34075 } }
I20260929 18:57:29.604853 19730 leader_election.cc:302] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5778636371b48be9151627adb85a5b5; no voters:
I20260929 18:57:29.605022 19730 leader_election.cc:288] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:57:29.605106 19739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:29.605247 19739 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [term 1 LEADER]: Becoming Leader. State: Replica: f5778636371b48be9151627adb85a5b5, State: Running, Role: LEADER
I20260929 18:57:29.605337 19730 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:29.605387 19739 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5778636371b48be9151627adb85a5b5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34075 } }
I20260929 18:57:29.605767 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5778636371b48be9151627adb85a5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5778636371b48be9151627adb85a5b5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34075 } } }
I20260929 18:57:29.605772 19742 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5778636371b48be9151627adb85a5b5. Latest consensus state: current_term: 1 leader_uuid: "f5778636371b48be9151627adb85a5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5778636371b48be9151627adb85a5b5" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 34075 } } }
I20260929 18:57:29.605985 19742 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:29.606206 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:29.606331 19749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:29.606889 19749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:29.607131 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:29.605044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33491 (local address 127.29.71.190:34075)
0929 18:57:29.605078 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:29.605080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:29.605849 (+ 769us) server_negotiation.cc:407] Connection header received
0929 18:57:29.605948 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:29.605949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:29.605954 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:29.605982 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:29.606696 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:29.606698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:29.606698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:57:29.606702 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:29.606714 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:57:29.606718 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:29.606746 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:29.606852 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:57:29.606865 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 18:57:29.608566 19749 catalog_manager.cc:1347] Generated new cluster ID: 422e07e856f745899c770c3f72e2ff21
I20260929 18:57:29.608608 19749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:29.620226 19749 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:29.620452 19749 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:29.626187 19749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5778636371b48be9151627adb85a5b5: Generated new TSK 0
I20260929 18:57:29.626418 19749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:29.632673 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19662
2026-09-29T18:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1348 ms)
[----------] 18 tests from AuthTokenIssuingTest (24073 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:29 dist-test-slave-q2kh krb5kdc[19761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:29 dist-test-slave-q2kh krb5kdc[19761](info): set up 2 sockets
Sep 29 18:57:29 dist-test-slave-q2kh krb5kdc[19761](info): commencing operation
krb5kdc: starting...
W20260929 18:57:30.670498 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:30 dist-test-slave-q2kh krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:30.789546 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:46175
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:46175,127.29.71.189:44033,127.29.71.188:35337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:30.868669 19774 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:57:30.868923 19774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:30.868952 19774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:30.868999 19774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:30.870184 19774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:30.870258 19774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:30.870282 19774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:30.870317 19774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:30.870348 19774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:30.871809 19774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:46175,127.29.71.189:44033,127.29.71.188:35337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:46175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:30.872103 19774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:30.872347 19774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:30.877213 19780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:30.877321 19782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:30.877308 19779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:30.877379 19774 server_base.cc:1034] running on GCE node
I20260929 18:57:30.877648 19774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:30.877957 19774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:30.879144 19774 hybrid_clock.cc:648] HybridClock initialized: now 1790708250879121 us; error 58 us; skew 500 ppm
Sep 29 18:57:30 dist-test-slave-q2kh krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:30.881712 19774 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:30.882206 19774 webserver.cc:460] Webserver started at http://127.29.71.190:45757/ using document root <none> and password file <none>
I20260929 18:57:30.882395 19774 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:30.882437 19774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:30.882524 19774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:30.883533 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "f81285ed199f486688a1c684db8ac375"
format_stamp: "Formatted at 2026-09-29 18:57:30 on dist-test-slave-q2kh"
server_key: "8d5777ba0aa4f573dd74835d8398f81c"
server_key_iv: "7a8abf1f7e95e768606dbd70d3c4fcf5"
server_key_version: "encryptionkey@0"
I20260929 18:57:30.883899 19774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "f81285ed199f486688a1c684db8ac375"
format_stamp: "Formatted at 2026-09-29 18:57:30 on dist-test-slave-q2kh"
server_key: "8d5777ba0aa4f573dd74835d8398f81c"
server_key_iv: "7a8abf1f7e95e768606dbd70d3c4fcf5"
server_key_version: "encryptionkey@0"
I20260929 18:57:30.885202 19774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:30.885950 19789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:30.886121 19774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:30.886245 19774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "f81285ed199f486688a1c684db8ac375"
format_stamp: "Formatted at 2026-09-29 18:57:30 on dist-test-slave-q2kh"
server_key: "8d5777ba0aa4f573dd74835d8398f81c"
server_key_iv: "7a8abf1f7e95e768606dbd70d3c4fcf5"
server_key_version: "encryptionkey@0"
I20260929 18:57:30.886327 19774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:30.898502 19774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:30.900808 19774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:30.900970 19774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:30.904985 19774 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:46175
I20260929 18:57:30.905035 19841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:46175 every 8 connection(s)
I20260929 18:57:30.905393 19774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:30.906361 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:30.907872 19842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:57:30 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:30.914351 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19774
I20260929 18:57:30.914459 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:30.914649 29982 external_mini_cluster.cc:1428] Setting key a77d5d90208edf59f75ea977a9b2d236
I20260929 18:57:30.917562 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:30.908550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.29.71.190:46175)
0929 18:57:30.908762 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:30.908766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:30.908799 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:57:30.909284 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:30.909286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:30.909320 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:30.909396 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:30.910044 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:30.910930 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:30.911760 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:30.912053 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:30.912130 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:30.914065 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:30.914071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:30.914071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:30.914077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:30.915930 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:30.916292 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:30.916295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:30.916295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:30.916417 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:30.916631 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:30.916632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:30.916633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:30.916932 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:30.917001 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:30.917189 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 18:57:30.917324 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260929 18:57:30.917564 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:30.908522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46175 (local address 127.0.0.1:48772)
0929 18:57:30.908702 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:30.908735 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:30.909160 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:30.909455 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:30.909458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:30.909748 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:30.909918 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:30.909922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:30.911022 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:30.911023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:30.911686 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:30.911688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:30.912151 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:30.912152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:30.912203 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:30.912649 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:30.912656 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:30.913991 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:30.916016 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:30.916022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:30.916024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:30.916212 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:30.916489 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:30.916489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:30.916490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:30.916566 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:30.916997 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:30.917002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:30.917054 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:57:30.917157 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:57:30.917384 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":94,"thread_start_us":90,"threads_started":1}
I20260929 18:57:30.918524 19842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:30.918869 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:30.918751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49306)
0929 18:57:30.918799 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:30.918845 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 18:57:30.919003 19790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
I20260929 18:57:30.919416 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:30.919326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49312)
0929 18:57:30.919372 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:30.919395 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:57:30.919519 19842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:44033: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.29.71.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab.
Entry for principal kudu/127.29.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab.
I20260929 18:57:30.941390 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:44033
--webserver_interface=127.29.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab
--principal=kudu/127.29.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:46175,127.29.71.189:44033,127.29.71.188:35337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:57:30.968753 19842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } attempt: 1
I20260929 18:57:30.969213 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:30.969083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49314)
0929 18:57:30.969143 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:30.969182 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:57:30.969573 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:30.969479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49330)
0929 18:57:30.969524 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:30.969548 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:57:30.969658 19842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:44033: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
W20260929 18:57:31.022920 19849 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:57:31.023147 19849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:31.023167 19849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:31.023190 19849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:31.024485 19849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:31.024533 19849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:31.024549 19849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:31.024561 19849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:31.024621 19849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:31.026144 19849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:46175,127.29.71.189:44033,127.29.71.188:35337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab
--principal=kudu/127.29.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:31.026448 19849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:31.026675 19849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:31.031683 19855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:31.031733 19854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:31.031718 19849 server_base.cc:1034] running on GCE node
W20260929 18:57:31.031652 19857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:31.032143 19849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:31.032452 19849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:31.033614 19849 hybrid_clock.cc:648] HybridClock initialized: now 1790708251033605 us; error 40 us; skew 500 ppm
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:31.036159 19849 init.cc:377] Logged in from keytab as kudu/127.29.71.189@KRBTEST.COM (short username kudu)
I20260929 18:57:31.036633 19849 webserver.cc:460] Webserver started at http://127.29.71.189:34347/ using document root <none> and password file <none>
I20260929 18:57:31.036823 19849 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:31.036863 19849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:31.036952 19849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:31.037809 19842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } attempt: 2
I20260929 18:57:31.037981 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data/instance:
uuid: "cd40d9e940924feb98383e6dd9cf00c4"
format_stamp: "Formatted at 2026-09-29 18:57:31 on dist-test-slave-q2kh"
server_key: "b26829e1ab05cc7f1e358cbc8f7ec80f"
server_key_iv: "16ace74a76b3a378120abb24b368d433"
server_key_version: "encryptionkey@0"
I20260929 18:57:31.038233 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.038076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49334)
0929 18:57:31.038143 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.038200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:57:31.038410 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/wal/instance:
uuid: "cd40d9e940924feb98383e6dd9cf00c4"
format_stamp: "Formatted at 2026-09-29 18:57:31 on dist-test-slave-q2kh"
server_key: "b26829e1ab05cc7f1e358cbc8f7ec80f"
server_key_iv: "16ace74a76b3a378120abb24b368d433"
server_key_version: "encryptionkey@0"
I20260929 18:57:31.038571 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.038493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49336)
0929 18:57:31.038521 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.038543 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260929 18:57:31.038666 19842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:44033: Network error: Client connection negotiation failed: client connection to 127.29.71.189:44033: connect: Connection refused (error 111)
I20260929 18:57:31.039796 19849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:31.040597 19864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:31.040858 19849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:31.040974 19849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/wal
uuid: "cd40d9e940924feb98383e6dd9cf00c4"
format_stamp: "Formatted at 2026-09-29 18:57:31 on dist-test-slave-q2kh"
server_key: "b26829e1ab05cc7f1e358cbc8f7ec80f"
server_key_iv: "16ace74a76b3a378120abb24b368d433"
server_key_version: "encryptionkey@0"
I20260929 18:57:31.041054 19849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:31.057050 19849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:31.059396 19849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:31.059558 19849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:31.063740 19849 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.189:44033
I20260929 18:57:31.063840 19916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.189:44033 every 8 connection(s)
I20260929 18:57:31.064133 19849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
I20260929 18:57:31.065064 19917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:31.066207 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19849
I20260929 18:57:31.066304 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-1/wal/instance
I20260929 18:57:31.066474 29982 external_mini_cluster.cc:1428] Setting key 984203cb812fe655341fa696a554e225
I20260929 18:57:31.066706 19917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.71.188@KRBTEST.COM; defaulting to no policy
I20260929 18:57:31.076092 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.067429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.29.71.190:46175)
0929 18:57:31.067469 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.067471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.067692 (+ 221us) server_negotiation.cc:407] Connection header received
0929 18:57:31.068188 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.068189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.068205 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.068259 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.068863 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.069349 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.070400 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.070601 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.070708 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.073040 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.073041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.073042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.073049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.074592 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.075096 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.075098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.075100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.075157 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.075409 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.075410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.075411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.075522 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.075602 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.075921 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.076001 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 18:57:31.076272 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.067396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46175 (local address 127.0.0.1:48776)
0929 18:57:31.067576 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.067609 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.068081 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.068329 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.068332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.068578 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.068770 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.068774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.069458 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.069459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.070315 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.070317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.070730 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.070731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.070871 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.071381 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.071389 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.072937 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.074686 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.074692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.074694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.075008 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.075222 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.075223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.075223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.075334 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.075591 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.075598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.075787 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.075889 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.076035 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.188@KRBTEST.COM" created.
I20260929 18:57:31.076808 19917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:31.084713 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.077095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49340)
0929 18:57:31.077147 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.077159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.077206 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.077492 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.077492 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.077599 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.077645 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.077646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.078709 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.078710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.079389 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.079392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.080076 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.080077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.080164 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.080575 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.080580 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.081683 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.083764 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.083765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.083766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.083931 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.084149 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.084150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.084150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.084196 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.084489 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.084490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.084522 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.084571 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.084649 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:57:31.084847 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.077178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.29.71.189:44033)
0929 18:57:31.077338 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.077343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.077367 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:57:31.077389 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.077392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.077406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.077482 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.077713 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.078617 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.079485 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.079997 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.080080 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.081765 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.081773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.081774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.081781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.083684 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.084014 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.084017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.084018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.084080 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.084250 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.084251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.084251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.084422 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.084511 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.084608 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.084727 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260929 18:57:31.085453 19917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:31.085906 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.085787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36196)
0929 18:57:31.085837 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.085878 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 18:57:31.086027 19868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
I20260929 18:57:31.086462 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.086338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36202)
0929 18:57:31.086417 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.086441 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 18:57:31.086584 19917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:35337: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab.
Entry for principal kudu/127.29.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab.
I20260929 18:57:31.093863 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:35337
--webserver_interface=127.29.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab
--principal=kudu/127.29.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:46175,127.29.71.189:44033,127.29.71.188:35337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:57:31.129822 19842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } attempt: 3
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:31.135812 19917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } attempt: 1
I20260929 18:57:31.136149 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.136047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36210)
0929 18:57:31.136094 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.136119 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:57:31.136495 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.136400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36212)
0929 18:57:31.136440 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.136461 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 18:57:31.136595 19917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:35337: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
I20260929 18:57:31.137069 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.130274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49348)
0929 18:57:31.130331 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.130349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.130420 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.130580 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.130581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.130746 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.130829 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.130831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.131507 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.131509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.132050 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.132052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.132474 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.132475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.132555 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.133023 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.133028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.134297 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.135985 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.135986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.135987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.136170 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.136427 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.136428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.136428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.136498 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.136836 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.136837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.136877 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.136938 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.137009 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:57:31.137141 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.130338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.29.71.189:44033)
0929 18:57:31.130401 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.130403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.130414 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:31.130489 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.130490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.130502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.130575 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.130921 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.131422 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.132148 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.132386 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.132473 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.134392 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.134393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.134394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.134400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.135846 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.136271 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.136272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.136272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.136342 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.136580 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.136581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.136582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.136725 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.136829 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.136985 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.137064 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:57:31.137575 19842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:31.137897 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.137803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36228)
0929 18:57:31.137847 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.137872 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 18:57:31.138620 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.138536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36236)
0929 18:57:31.138582 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.138605 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:57:31.138726 19842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:35337: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
I20260929 18:57:31.169852 19842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } attempt: 1
I20260929 18:57:31.170356 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.170221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36248)
0929 18:57:31.170287 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.170328 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:57:31.170713 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.170634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36264)
0929 18:57:31.170676 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.170699 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:57:31.170818 19842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:35337: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
W20260929 18:57:31.177481 19924 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:57:31.177678 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:31.177695 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:31.177744 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:31.178946 19924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:31.178987 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:31.179001 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:31.179045 19924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:31.179068 19924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:31.180549 19924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:46175,127.29.71.189:44033,127.29.71.188:35337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab
--principal=kudu/127.29.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:35337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:31.180809 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:31.181043 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:31.185778 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:31.185804 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:31.185748 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:31.186079 19924 server_base.cc:1034] running on GCE node
I20260929 18:57:31.186371 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:31.186677 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:31.187846 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790708251187822 us; error 36 us; skew 500 ppm
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:31.190595 19924 init.cc:377] Logged in from keytab as kudu/127.29.71.188@KRBTEST.COM (short username kudu)
I20260929 18:57:31.191035 19924 webserver.cc:460] Webserver started at http://127.29.71.188:38687/ using document root <none> and password file <none>
I20260929 18:57:31.191277 19924 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:31.191352 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:31.191501 19924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:31.192658 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data/instance:
uuid: "9c86abbde39c4407a18c1a831522d54d"
format_stamp: "Formatted at 2026-09-29 18:57:31 on dist-test-slave-q2kh"
server_key: "ca28b43ddc80901db8c69b165cb4ce0a"
server_key_iv: "fd55830d9f1d48470dee53a18ea61930"
server_key_version: "encryptionkey@0"
I20260929 18:57:31.193017 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/wal/instance:
uuid: "9c86abbde39c4407a18c1a831522d54d"
format_stamp: "Formatted at 2026-09-29 18:57:31 on dist-test-slave-q2kh"
server_key: "ca28b43ddc80901db8c69b165cb4ce0a"
server_key_iv: "fd55830d9f1d48470dee53a18ea61930"
server_key_version: "encryptionkey@0"
I20260929 18:57:31.194346 19924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 18:57:31.195057 19939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:31.195221 19924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 18:57:31.195329 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/wal
uuid: "9c86abbde39c4407a18c1a831522d54d"
format_stamp: "Formatted at 2026-09-29 18:57:31 on dist-test-slave-q2kh"
server_key: "ca28b43ddc80901db8c69b165cb4ce0a"
server_key_iv: "fd55830d9f1d48470dee53a18ea61930"
server_key_version: "encryptionkey@0"
I20260929 18:57:31.195401 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:31.204728 19917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } attempt: 2
I20260929 18:57:31.205204 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.205074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36272)
0929 18:57:31.205131 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.205171 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 18:57:31.205592 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.205516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36282)
0929 18:57:31.205554 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.205576 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 18:57:31.205708 19917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:35337: Network error: Client connection negotiation failed: client connection to 127.29.71.188:35337: connect: Connection refused (error 111)
I20260929 18:57:31.216496 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:31.218855 19924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:31.219017 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:31.222831 19924 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.188:35337
I20260929 18:57:31.222903 19991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.188:35337 every 8 connection(s)
I20260929 18:57:31.223239 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
I20260929 18:57:31.224216 19992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:31.225711 19992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:31.228466 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 19924
I20260929 18:57:31.228569 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790707954762284-29982-0/minicluster-data/master-2/wal/instance
I20260929 18:57:31.228737 29982 external_mini_cluster.cc:1428] Setting key e0029e17f6aaba3792ecb13c769ee420
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:57:31.235142 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.226435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.29.71.190:46175)
0929 18:57:31.226502 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.226504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.226739 (+ 235us) server_negotiation.cc:407] Connection header received
0929 18:57:31.227166 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.227167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.227176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.227232 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.227770 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.228232 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.229483 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.229810 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.229894 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.232294 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.232295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.232296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.232302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.233734 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.234192 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.234194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.234194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.234256 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.234497 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.234498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.234498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.234639 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.234707 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.234993 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.235068 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:57:31.235455 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.226427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:46175 (local address 127.0.0.1:48782)
0929 18:57:31.226594 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.226636 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.227076 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.227282 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.227285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.227517 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.227700 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.227704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.228327 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.228328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.229370 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.229372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.229934 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.229936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.230047 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.230538 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.230547 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.232176 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.233815 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.233822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.233824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.234106 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.234328 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.234329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.234329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.234431 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.234714 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.234719 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.234857 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.234943 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.235204 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260929 18:57:31.236012 19992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:31.238107 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.229125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.29.71.188:35337)
0929 18:57:31.229271 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.229276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.229322 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:57:31.229366 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.229368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.229391 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.229463 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.230221 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.231284 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.232237 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.232552 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.232634 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.235120 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.235126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.235126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.235132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.236990 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.237288 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.237290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.237291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.237348 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.237526 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.237527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.237527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.237671 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.237831 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.237912 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.237990 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:31.242777 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.236261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:44033 (local address 127.0.0.1:49350)
0929 18:57:31.236320 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.236333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.236373 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.236549 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.236549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.236669 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.236696 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.236697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.237491 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.237492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.237968 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.237969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.238444 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.238445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.238524 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.238920 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.238924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.240225 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.241785 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.241812 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.241812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.242030 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.242234 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.242235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.242235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.242296 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.242574 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.242575 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.242607 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.242656 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.242720 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 18:57:31.242905 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.236346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.29.71.189:44033)
0929 18:57:31.236435 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.236437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.236446 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:57:31.236471 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.236472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.236482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.236547 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.236764 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.237426 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.238048 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.238372 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.238446 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.240311 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.240312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.240312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.240318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.241706 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.242099 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.242100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.242100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.242153 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.242362 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.242363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.242364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.242503 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.242581 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.242684 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.242763 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:57:31.243211 19992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:31.245954 19842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } attempt: 2
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:57:31.247114 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.239027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.29.71.190:46175)
0929 18:57:31.239070 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.239072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.239080 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:57:31.239120 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.239121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.239130 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.239183 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.239761 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.240363 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.241367 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.241544 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.241592 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.244284 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.244285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.244285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.244292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.245843 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.246307 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.246308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.246308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.246367 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.246577 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.246579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.246579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.246709 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.246898 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.246937 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.247008 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:57:31.250556 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.243446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36300)
0929 18:57:31.243492 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.243501 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.243537 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.243687 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.243687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.243858 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.243888 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.243890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.244634 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.244635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.245157 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.245158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.245544 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.245544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.245592 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.245967 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.245971 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.247507 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.249119 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.249120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.249120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.249333 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.249528 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.249529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.249529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.249581 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.250229 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.250229 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.250270 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.250356 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.250452 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 18:57:31.250934 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.243493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.29.71.188:35337)
0929 18:57:31.243532 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.243534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.243543 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:57:31.243582 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.243583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.243593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.243686 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.243964 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.244558 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.245243 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.245463 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.245543 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.247602 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.247603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.247603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.247610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.249001 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.249403 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.249404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.249404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.249453 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.249661 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.249661 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.249662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.249759 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.249845 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.250392 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.250536 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":19}
I20260929 18:57:31.252480 19992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: Bootstrap starting.
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:31.253180 19992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:31.253476 19992 log.cc:824] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:31.254190 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.246197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36306)
0929 18:57:31.246239 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.246248 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.246281 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.246530 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.246531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.246677 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.246710 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.246712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.247638 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.247639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.248441 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.248443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.248925 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.248927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.248979 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.249335 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.249340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.250725 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.253004 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.253005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.253006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.253228 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.253436 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.253437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.253437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.253500 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.253832 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.253833 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.253880 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.253963 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.254074 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 18:57:31.254273 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.246248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.29.71.188:35337)
0929 18:57:31.246401 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.246403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.246410 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:57:31.246427 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.246428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.246438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.246796 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.246832 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.247549 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.248524 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.248847 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.249109 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.250809 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.250810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.250810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.250816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.252908 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.253309 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.253309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.253310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.253363 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.253575 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.253576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.253576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.253750 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.253826 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.254003 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.254085 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260929 18:57:31.254724 19992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: No bootstrap required, opened a new log
I20260929 18:57:31.256101 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.247708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.29.71.189:44033)
0929 18:57:31.247739 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.247741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.247750 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:57:31.247767 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.247768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.247776 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.247823 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.248389 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.249057 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.249969 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.250312 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.250377 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.253309 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.253310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.253310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.253317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.255044 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.255361 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.255362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.255362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.255422 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.255586 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.255587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.255587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.255714 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.255885 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.255937 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.256014 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 18:57:31.256904 19992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.257081 19992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c86abbde39c4407a18c1a831522d54d, State: Initialized, Role: FOLLOWER
I20260929 18:57:31.257180 19992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.257460 19992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:31.257432 19999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.257551 19999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 18:57:31.259296 20010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:31.259397 20010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:31.259868 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375: Bootstrap starting.
I20260929 18:57:31.260569 19842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:31.260874 19842 log.cc:824] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:31.261560 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:31.259397 19924 master_runner.cc:186] Error getting master registration for 127.29.71.190:46175: OK, instance_id { permanent_uuid: "f81285ed199f486688a1c684db8ac375" instance_seqno: 1790708250903750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:57:31.261922 19924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:57:31.262739 19842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375: No bootstrap required, opened a new log
I20260929 18:57:31.262566 19816 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f81285ed199f486688a1c684db8ac375"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48790
I20260929 18:57:31.264437 19842 raft_consensus.cc:348] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.264580 19842 raft_consensus.cc:729] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81285ed199f486688a1c684db8ac375, State: Initialized, Role: FOLLOWER
I20260929 18:57:31.264663 19842 consensus_queue.cc:260] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.264973 19842 sys_catalog.cc:564] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:31.265027 19816 raft_consensus.cc:482] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:57:31.265157 19816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:31.265952 19816 raft_consensus.cc:504] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.266245 19816 leader_election.cc:288] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [CANDIDATE]: Term 1 election: Requested vote from peers cd40d9e940924feb98383e6dd9cf00c4 (127.29.71.189:44033), 9c86abbde39c4407a18c1a831522d54d (127.29.71.188:35337)
I20260929 18:57:31.266585 19966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f81285ed199f486688a1c684db8ac375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c86abbde39c4407a18c1a831522d54d"
I20260929 18:57:31.266706 19966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:57:31.267024 20024 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:31.267069 20024 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:31.267344 20013 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.267419 20013 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:31.268013 19966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f81285ed199f486688a1c684db8ac375 in term 1.
I20260929 18:57:31.268261 19793 leader_election.cc:302] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c86abbde39c4407a18c1a831522d54d, f81285ed199f486688a1c684db8ac375; no voters:
I20260929 18:57:31.268457 20013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:31.268656 20013 raft_consensus.cc:686] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [term 1 LEADER]: Becoming Leader. State: Replica: f81285ed199f486688a1c684db8ac375, State: Running, Role: LEADER
I20260929 18:57:31.268793 20013 consensus_queue.cc:237] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.269269 20013 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f81285ed199f486688a1c684db8ac375. Latest consensus state: current_term: 1 leader_uuid: "f81285ed199f486688a1c684db8ac375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.269428 20013 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:31.267824 19774 master_runner.cc:186] Error getting master registration for 127.29.71.189:44033: OK, instance_id { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" instance_seqno: 1790708251062606 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:57:31.270632 19774 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:57:31.271354 20033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:31.271600 20033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:31.273137 19966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [term 1 FOLLOWER]: Refusing update from remote peer f81285ed199f486688a1c684db8ac375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:31.273487 20026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:31.274803 19999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f81285ed199f486688a1c684db8ac375. Latest consensus state: current_term: 1 leader_uuid: "f81285ed199f486688a1c684db8ac375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.274883 19999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:31.277976 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.269245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.29.71.190:46175)
0929 18:57:31.269297 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.269299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.269880 (+ 581us) server_negotiation.cc:407] Connection header received
0929 18:57:31.269982 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.269983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.269993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.270023 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.270859 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.271591 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.272676 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.272964 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.273004 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.274766 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.274767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.274768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.274775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.276603 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.276996 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.276997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.276997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.277051 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.277258 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.277259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.277259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.277393 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.277476 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.277691 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.277765 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:31.281098 20013 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f81285ed199f486688a1c684db8ac375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.281214 20013 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:31.281289 19999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f81285ed199f486688a1c684db8ac375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.281375 19999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:31.281486 20036 mvcc.cc:205] Tried to move back new op lower bound from 7334740997212667904 to 7334740997198000128. Current Snapshot: MvccSnapshot[applied={T|T < 7334740997212667904}]
I20260929 18:57:31.281971 20037 mvcc.cc:205] Tried to move back new op lower bound from 7334740997212667904 to 7334740997198000128. Current Snapshot: MvccSnapshot[applied={T|T < 7334740997212667904}]
I20260929 18:57:31.282945 20033 catalog_manager.cc:1347] Generated new cluster ID: 88943a0d4f9c431485e1089452746a78
I20260929 18:57:31.282994 20033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:31.283710 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.269645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.29.71.188:35337)
0929 18:57:31.269691 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.269693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.270829 (+ 1136us) server_negotiation.cc:407] Connection header received
0929 18:57:31.270937 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.270938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.270949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.271001 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.273790 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.274498 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.276225 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.276576 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.276635 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.279741 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.279743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.279743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.279751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.282360 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.282729 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.282730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.282730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.282780 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.282966 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.282966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.282967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.283094 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.283174 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.283549 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.283623 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:57:31.286671 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.270245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.29.71.189:44033)
0929 18:57:31.270287 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.270289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.271007 (+ 718us) server_negotiation.cc:407] Connection header received
0929 18:57:31.271104 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.271105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.271116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.271145 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.272979 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.273666 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.276885 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.277187 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.277257 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.279187 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.279188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.279189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.279222 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.281448 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.281953 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.281954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.281954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.282014 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.282497 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.282498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.282498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.282669 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.282765 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.286501 (+ 3736us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.286582 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:57:31.296871 19917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } attempt: 3
Sep 29 18:57:31 dist-test-slave-q2kh krb5kdc[19761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708251, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:57:31.304718 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.297124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:35337 (local address 127.0.0.1:36328)
0929 18:57:31.297173 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:31.297184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:31.297229 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:31.297354 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:31.297355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:31.297532 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:31.297575 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.297577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.298482 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.298483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.299293 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:31.299295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.299746 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.299747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:31.299801 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.300293 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:31.300297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:31.301962 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:31.303747 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.303748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.303748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.303934 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.304120 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:31.304121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:31.304121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:31.304177 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:31.304413 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:31.304414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:31.304458 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 18:57:31.304523 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:57:31.304619 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 18:57:31.304718 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:31.297180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.29.71.188:35337)
0929 18:57:31.297234 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.297236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.297244 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:57:31.297296 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.297297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.297306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.297350 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.297639 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.298417 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.299366 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:31.299672 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:31.299726 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:31.302048 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:31.302049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:31.302050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:31.302058 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:31.303659 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.303998 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.303999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.303999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.304051 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:31.304248 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:31.304248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:31.304249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:31.304366 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:31.304423 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:31.304546 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 18:57:31.304619 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 18:57:31.306325 19917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: Bootstrap starting.
I20260929 18:57:31.307065 19917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:31.307420 19917 log.cc:824] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:31.309255 19917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: No bootstrap required, opened a new log
I20260929 18:57:31.311360 19917 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.311501 19917 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd40d9e940924feb98383e6dd9cf00c4, State: Initialized, Role: FOLLOWER
I20260929 18:57:31.311589 19917 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } }
I20260929 18:57:31.311902 20042 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.312005 19891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f81285ed199f486688a1c684db8ac375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd40d9e940924feb98383e6dd9cf00c4"
I20260929 18:57:31.311985 19890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:31.312973 19890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [term 1 FOLLOWER]: Refusing update from remote peer f81285ed199f486688a1c684db8ac375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:31.313305 20013 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:31.313845 19891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f81285ed199f486688a1c684db8ac375 in term 1.
I20260929 18:57:31.311931 19917 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:31.315219 20048 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f81285ed199f486688a1c684db8ac375. Latest consensus state: current_term: 1 leader_uuid: "f81285ed199f486688a1c684db8ac375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.315354 20048 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:31.315459 20048 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f81285ed199f486688a1c684db8ac375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.315572 20048 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:31.316793 20044 mvcc.cc:205] Tried to move back new op lower bound from 7334740997212667904 to 7334740997198000128. Current Snapshot: MvccSnapshot[applied={T|T < 7334740997212667904}]
I20260929 18:57:31.312151 20042 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:31.319638 20013 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f81285ed199f486688a1c684db8ac375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81285ed199f486688a1c684db8ac375" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 46175 } } peers { permanent_uuid: "cd40d9e940924feb98383e6dd9cf00c4" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 44033 } } peers { permanent_uuid: "9c86abbde39c4407a18c1a831522d54d" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 35337 } } }
I20260929 18:57:31.319753 20013 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:31.320040 20057 catalog_manager.cc:1259] Loaded cluster ID: 88943a0d4f9c431485e1089452746a78
I20260929 18:57:31.320083 20057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: loading cluster ID for follower catalog manager: success
W20260929 18:57:31.320165 20057 catalog_manager.cc:1576] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260929 18:57:31.320214 20057 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:31.321209 20033 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:31.321906 20033 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:31.338301 20033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f81285ed199f486688a1c684db8ac375: Generated new TSK 0
I20260929 18:57:31.338539 20033 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:57:31.361529 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.360507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.29.71.189:44033)
0929 18:57:31.360554 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.360556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.360832 (+ 276us) server_negotiation.cc:407] Connection header received
0929 18:57:31.360943 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.360944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.360955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.360986 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.361460 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49390: BlockingRecv error: recv got EOF from 127.0.0.1:49390 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 18:57:31.362658 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.361956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.29.71.190:46175)
0929 18:57:31.361997 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.361998 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.362191 (+ 193us) server_negotiation.cc:407] Connection header received
0929 18:57:31.362227 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.362228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.362238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.362284 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.362596 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48810: BlockingRecv error: recv got EOF from 127.0.0.1:48810 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 18:57:31.363555 19997 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.362914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.29.71.188:35337)
0929 18:57:31.362954 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.362956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.363114 (+ 158us) server_negotiation.cc:407] Connection header received
0929 18:57:31.363139 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.363140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.363148 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.363190 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.363496 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36332: BlockingRecv error: recv got EOF from 127.0.0.1:36332 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 18:57:31.365801 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.365029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.29.71.189:44033)
0929 18:57:31.365064 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.365065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.365296 (+ 231us) server_negotiation.cc:407] Connection header received
0929 18:57:31.365405 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.365406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.365423 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.365451 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.365755 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49406: BlockingRecv error: recv got EOF from 127.0.0.1:49406 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 18:57:31.368698 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.367992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.29.71.189:44033)
0929 18:57:31.368024 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.368026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.368247 (+ 221us) server_negotiation.cc:407] Connection header received
0929 18:57:31.368340 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.368341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.368350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.368377 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.368655 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49418: BlockingRecv error: recv got EOF from 127.0.0.1:49418 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 18:57:31.372563 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.371749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.29.71.189:44033)
0929 18:57:31.371791 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.371793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.372086 (+ 293us) server_negotiation.cc:407] Connection header received
0929 18:57:31.372176 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.372177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.372186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.372224 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.372520 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49428: BlockingRecv error: recv got EOF from 127.0.0.1:49428 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 18:57:31.378497 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.377656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.29.71.189:44033)
0929 18:57:31.377690 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.377691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.377992 (+ 301us) server_negotiation.cc:407] Connection header received
0929 18:57:31.378084 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.378085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.378094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.378121 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.378454 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49440: BlockingRecv error: recv got EOF from 127.0.0.1:49440 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 18:57:31.388679 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.387771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.29.71.189:44033)
0929 18:57:31.387809 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.387811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.388107 (+ 296us) server_negotiation.cc:407] Connection header received
0929 18:57:31.388202 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.388203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.388214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.388245 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.388618 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49454: BlockingRecv error: recv got EOF from 127.0.0.1:49454 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 18:57:31.407027 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.406075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.29.71.189:44033)
0929 18:57:31.406117 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.406119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.406422 (+ 303us) server_negotiation.cc:407] Connection header received
0929 18:57:31.406532 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.406533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.406545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.406576 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.406968 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49462: BlockingRecv error: recv got EOF from 127.0.0.1:49462 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 18:57:31.441414 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.440510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.29.71.189:44033)
0929 18:57:31.440557 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.440559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.440819 (+ 260us) server_negotiation.cc:407] Connection header received
0929 18:57:31.440924 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.440925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.440936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.440967 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.441351 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49472: BlockingRecv error: recv got EOF from 127.0.0.1:49472 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 18:57:31.507813 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.507165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.29.71.189:44033)
0929 18:57:31.507211 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.507213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.507223 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:31.507254 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.507255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.507267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.507327 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.507747 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49480: BlockingRecv error: recv got EOF from 127.0.0.1:49480 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 18:57:31.638024 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.637223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.29.71.189:44033)
0929 18:57:31.637316 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.637319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.637416 (+ 97us) server_negotiation.cc:407] Connection header received
0929 18:57:31.637516 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.637517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.637529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.637603 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.637961 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49490: BlockingRecv error: recv got EOF from 127.0.0.1:49490 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 18:57:31.895869 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:31.895199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.29.71.189:44033)
0929 18:57:31.895250 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:31.895252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:31.895262 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:31.895358 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:31.895359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:31.895377 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:31.895442 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:31.895803 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49496: BlockingRecv error: recv got EOF from 127.0.0.1:49496 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:57:32.259932 20010 catalog_manager.cc:1259] Loaded cluster ID: 88943a0d4f9c431485e1089452746a78
I20260929 18:57:32.260000 20010 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: loading cluster ID for follower catalog manager: success
I20260929 18:57:32.260769 20010 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: acquiring CA information for follower catalog manager: success
I20260929 18:57:32.261061 20010 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c86abbde39c4407a18c1a831522d54d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:57:32.321280 20057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: acquiring CA information for follower catalog manager: success
I20260929 18:57:32.321595 20057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd40d9e940924feb98383e6dd9cf00c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:57:32.411872 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:32.409140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.29.71.189:44033)
0929 18:57:32.409273 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:32.409275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:32.409323 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:57:32.409360 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:32.409361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:32.409430 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:32.409519 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:32.409660 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.410237 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.411049 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.411262 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.411348 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:32.411512 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:32.411629 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:32.411715 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:32.411757 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 18:57:32.411776 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260929 18:57:32.416621 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:32.413845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.29.71.188:35337)
0929 18:57:32.413968 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:32.413970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:32.414001 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:57:32.414043 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:32.414043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:32.414089 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:32.414216 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:32.414341 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.414995 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.415723 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.416018 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.416092 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:32.416259 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:32.416375 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:32.416473 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:32.416512 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 18:57:32.416533 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":81,"threads_started":1}
I20260929 18:57:32.421142 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:32.418197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.29.71.189:44033)
0929 18:57:32.418260 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:32.418262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:32.418451 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:57:32.418535 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:32.418536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:32.418548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:32.418607 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:32.418719 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.419293 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.420323 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.420570 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.420643 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:32.420812 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:32.420845 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:32.420892 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:32.421051 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 18:57:32.421072 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:57:32.421294 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:32.418510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.29.71.188:35337)
0929 18:57:32.418656 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:32.418657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:32.418767 (+ 110us) server_negotiation.cc:407] Connection header received
0929 18:57:32.418827 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:32.418828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:32.418837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:32.418885 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:32.418977 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.419653 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.420460 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.420735 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.420803 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:32.420962 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:32.420992 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:32.421039 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:32.421185 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:57:32.421205 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 18:57:32.421605 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:32.418195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.29.71.190:46175)
0929 18:57:32.418314 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:32.418316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:32.418434 (+ 118us) server_negotiation.cc:407] Connection header received
0929 18:57:32.418476 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:32.418477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:32.418525 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:32.418601 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:32.418730 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.419508 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.420494 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:32.420796 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:32.420864 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:32.421049 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:32.421149 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:32.421208 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:32.421421 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 18:57:32.421446 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260929 18:57:32.423363 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19774
I20260929 18:57:32.428463 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19849
I20260929 18:57:32.433007 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 19924
2026-09-29T18:57:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2801 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:57:32 dist-test-slave-q2kh krb5kdc[20141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:57:32 dist-test-slave-q2kh krb5kdc[20141](info): set up 2 sockets
Sep 29 18:57:32 dist-test-slave-q2kh krb5kdc[20141](info): commencing operation
krb5kdc: starting...
W20260929 18:57:33.470448 29982 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/test-user.keytab.
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.71.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
Entry for principal kudu/127.29.71.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab.
I20260929 18:57:33.594555 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:40933
--webserver_interface=127.29.71.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:40933,127.29.71.189:42229,127.29.71.188:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:57:33.678014 20154 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:57:33.678290 20154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:33.678323 20154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:33.678368 20154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:33.679497 20154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:33.679571 20154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:33.679600 20154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:33.679638 20154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:33.679673 20154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:33.681154 20154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:40933,127.29.71.189:42229,127.29.71.188:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.190.keytab
--principal=kudu/127.29.71.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.190:40933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.71.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:33.681450 20154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:33.681699 20154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:33.686582 20162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:33.686833 20154 server_base.cc:1034] running on GCE node
W20260929 18:57:33.686583 20159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:33.686570 20160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:33.687134 20154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:33.687443 20154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:33.688638 20154 hybrid_clock.cc:648] HybridClock initialized: now 1790708253688623 us; error 24 us; skew 500 ppm
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:33.691287 20154 init.cc:377] Logged in from keytab as kudu/127.29.71.190@KRBTEST.COM (short username kudu)
I20260929 18:57:33.691830 20154 webserver.cc:460] Webserver started at http://127.29.71.190:35963/ using document root <none> and password file <none>
I20260929 18:57:33.692081 20154 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:33.692154 20154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:33.692317 20154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:33.693456 20154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data/instance:
uuid: "0c9bc6a31ddf4e8daa396483900ccbe6"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "8f53318e9cca8418c1e5f7dec34b68b2"
server_key_iv: "edfe085db194a64efad4efa26c341951"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.693851 20154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/wal/instance:
uuid: "0c9bc6a31ddf4e8daa396483900ccbe6"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "8f53318e9cca8418c1e5f7dec34b68b2"
server_key_iv: "edfe085db194a64efad4efa26c341951"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.695341 20154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:33.696256 20169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:33.696463 20154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:33.696584 20154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/wal
uuid: "0c9bc6a31ddf4e8daa396483900ccbe6"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "8f53318e9cca8418c1e5f7dec34b68b2"
server_key_iv: "edfe085db194a64efad4efa26c341951"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.696669 20154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:33.705829 20154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:33.707868 20154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:33.708069 20154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:33.712532 20154 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.190:40933
I20260929 18:57:33.712564 20221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.190:40933 every 8 connection(s)
I20260929 18:57:33.712962 20154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/data/info.pb
I20260929 18:57:33.713945 20222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:33.715533 20222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:33.719203 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 20154
I20260929 18:57:33.719310 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-0/wal/instance
I20260929 18:57:33.719496 29982 external_mini_cluster.cc:1428] Setting key a5791ba4b6e0ae32ebcfddf4e9614298
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:33.726830 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.716292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:40933 (local address 127.0.0.1:36728)
0929 18:57:33.716610 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.716653 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:33.717152 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:33.717441 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:33.717443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:33.717721 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:33.717871 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.717875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.718955 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.718956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.720062 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.720065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.720616 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.720617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.720701 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.721358 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:33.721366 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:33.722789 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:33.725236 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.725242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.725244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.725438 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.725745 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.725746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.725746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.725835 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.726361 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:33.726367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:33.726426 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 18:57:33.726503 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:57:33.726633 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"spinlock_wait_cycles":8064,"thread_start_us":178,"threads_started":1}
I20260929 18:57:33.726910 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.716293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.29.71.190:40933)
0929 18:57:33.716593 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:33.716611 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:33.716759 (+ 148us) server_negotiation.cc:407] Connection header received
0929 18:57:33.717312 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:33.717316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:33.717339 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:33.717436 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:33.717969 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.718839 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.720155 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.720500 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.720598 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.722895 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:33.722904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:33.722904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:33.722913 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:33.725124 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.725527 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.725531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.725531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.725656 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.725918 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.725919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.725919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.726282 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:33.726390 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:33.726578 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 18:57:33.726741 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":234,"thread_start_us":155,"threads_started":1}
I20260929 18:57:33.727666 20222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:33.728026 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.727902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35796)
0929 18:57:33.727950 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.727993 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:57:33.728145 20173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.29.71.189@KRBTEST.COM; defaulting to no policy
I20260929 18:57:33.728606 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.728480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35800)
0929 18:57:33.728539 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.728578 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 18:57:33.728761 20222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:42229: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab.
Entry for principal kudu/127.29.71.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab.
I20260929 18:57:33.748764 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:42229
--webserver_interface=127.29.71.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab
--principal=kudu/127.29.71.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:40933,127.29.71.189:42229,127.29.71.188:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:57:33.778048 20222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } attempt: 1
I20260929 18:57:33.778692 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.778516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35812)
0929 18:57:33.778597 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.778653 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:57:33.779153 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.779052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35822)
0929 18:57:33.779101 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.779131 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 18:57:33.779260 20222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:42229: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
W20260929 18:57:33.834210 20229 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:57:33.834436 20229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:33.834455 20229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:33.834478 20229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:33.835662 20229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:33.835707 20229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:33.835724 20229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:33.835739 20229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:33.835788 20229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:33.837324 20229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:40933,127.29.71.189:42229,127.29.71.188:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.189.keytab
--principal=kudu/127.29.71.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.189:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.71.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:33.837602 20229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:33.837862 20229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:33.842753 20234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:33.842846 20235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:33.842801 20237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:33.843120 20229 server_base.cc:1034] running on GCE node
I20260929 18:57:33.843375 20229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:33.843613 20229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:33.844795 20229 hybrid_clock.cc:648] HybridClock initialized: now 1790708253844785 us; error 30 us; skew 500 ppm
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:33.847167 20229 init.cc:377] Logged in from keytab as kudu/127.29.71.189@KRBTEST.COM (short username kudu)
I20260929 18:57:33.847407 20222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } attempt: 2
I20260929 18:57:33.847636 20229 webserver.cc:460] Webserver started at http://127.29.71.189:34563/ using document root <none> and password file <none>
I20260929 18:57:33.847834 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.847715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35838)
0929 18:57:33.847766 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.847805 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 18:57:33.847920 20229 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:33.847983 20229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:33.848132 20229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:33.848213 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.848098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35854)
0929 18:57:33.848147 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.848174 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 18:57:33.848294 20222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.189:42229: Network error: Client connection negotiation failed: client connection to 127.29.71.189:42229: connect: Connection refused (error 111)
I20260929 18:57:33.849298 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data/instance:
uuid: "54fdd297eddd45939bef0c88a60ef353"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "a541c2e7618c20c267767c07485eb157"
server_key_iv: "07ea273a99eb6131e70b1d2c1e8c2774"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.849649 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/wal/instance:
uuid: "54fdd297eddd45939bef0c88a60ef353"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "a541c2e7618c20c267767c07485eb157"
server_key_iv: "07ea273a99eb6131e70b1d2c1e8c2774"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.851130 20229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 18:57:33.852005 20244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:33.852267 20229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 18:57:33.852339 20229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/wal
uuid: "54fdd297eddd45939bef0c88a60ef353"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "a541c2e7618c20c267767c07485eb157"
server_key_iv: "07ea273a99eb6131e70b1d2c1e8c2774"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.852425 20229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:33.864274 20229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:33.866144 20229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:33.866367 20229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:33.870393 20229 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.189:42229
I20260929 18:57:33.870463 20296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.189:42229 every 8 connection(s)
I20260929 18:57:33.870790 20229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/data/info.pb
I20260929 18:57:33.871661 20297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:33.873193 20297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:33.873685 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 20229
I20260929 18:57:33.873768 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-1/wal/instance
I20260929 18:57:33.874011 29982 external_mini_cluster.cc:1428] Setting key 8f6be8cd4ba60ae84d5c562d62749b7d
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.71.188@KRBTEST.COM; defaulting to no policy
I20260929 18:57:33.882987 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.873895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.29.71.190:40933)
0929 18:57:33.873954 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:33.873957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:33.874234 (+ 277us) server_negotiation.cc:407] Connection header received
0929 18:57:33.874715 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:33.874717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:33.874735 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:33.874817 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:33.875520 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.876008 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.877096 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.877455 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.877538 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.879938 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:33.879939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:33.879939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:33.879948 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:33.881592 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.882039 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.882041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.882041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.882102 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.882384 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.882385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.882385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.882486 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:33.882534 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:33.882770 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:57:33.882887 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 18:57:33.883217 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.873893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:40933 (local address 127.0.0.1:36736)
0929 18:57:33.874075 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.874117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:33.874616 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:33.874875 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:33.874878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:33.875120 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:33.875419 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.875426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.876108 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.876109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.877001 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.877004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.877559 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.877560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.877668 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.878249 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:33.878258 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:33.879842 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:33.881674 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.881680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.881682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.881957 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.882240 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.882241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.882241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.882340 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.882539 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:33.882544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:33.882653 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 18:57:33.882734 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:57:33.882858 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260929 18:57:33.883880 20297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.71.188@KRBTEST.COM" created.
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:33.891606 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.884166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.29.71.189:42229)
0929 18:57:33.884321 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:33.884325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:33.884344 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:57:33.884368 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:33.884370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:33.884383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:33.884877 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:33.885135 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.885810 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.886559 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.886880 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.886968 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.888765 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:33.888772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:33.888772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:33.888778 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:33.890373 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.890757 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.890759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.890760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.890837 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.891044 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.891045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.891045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.891168 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:33.891229 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:33.891391 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:57:33.891508 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260929 18:57:33.891606 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.884119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35866)
0929 18:57:33.884185 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.884195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:33.884234 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:33.884879 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:33.884880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:33.885017 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:33.885058 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.885059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.885914 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.885915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.886470 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.886472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.886967 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.886968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.887030 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.887455 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:33.887460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:33.888676 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:33.890454 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.890456 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.890456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.890679 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.890908 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.890909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.890910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.890977 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.891234 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:33.891235 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:33.891282 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:57:33.891357 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:57:33.891461 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:57:33.892295 20297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:33.892670 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.892568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38928)
0929 18:57:33.892611 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.892652 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 18:57:33.892874 20246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
I20260929 18:57:33.893252 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.893177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38934)
0929 18:57:33.893214 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.893238 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 18:57:33.893373 20297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:45569: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.71.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab.
Entry for principal kudu/127.29.71.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab.
I20260929 18:57:33.900527 29982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWZtMeW/build/release/bin/kudu
/tmp/dist-test-taskWZtMeW/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:45569
--webserver_interface=127.29.71.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab
--principal=kudu/127.29.71.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.71.148:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.71.190:40933,127.29.71.189:42229,127.29.71.188:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:57:33.939432 20222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } attempt: 3
I20260929 18:57:33.942623 20297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } attempt: 1
I20260929 18:57:33.943007 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.942873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38950)
0929 18:57:33.942931 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.942970 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:57:33.943363 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.943243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38958)
0929 18:57:33.943299 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.943333 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:57:33.943454 20297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:45569: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:33.946568 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.939777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35878)
0929 18:57:33.939843 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.939862 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:33.939928 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:33.940080 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:33.940081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:33.940242 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:33.940322 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.940324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.941041 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.941042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.941575 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:33.941576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.941977 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.941978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:33.942057 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.942545 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:33.942567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:33.944010 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:33.945660 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.945661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.945662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.945828 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.946025 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:33.946026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:33.946026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:33.946074 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:33.946338 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:33.946339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:33.946377 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:57:33.946431 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 18:57:33.946504 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:57:33.946630 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.939861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.29.71.189:42229)
0929 18:57:33.939929 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:33.939932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:33.939943 (+ 11us) server_negotiation.cc:407] Connection header received
0929 18:57:33.939992 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:33.939993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:33.940005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:33.940076 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:33.940410 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.940928 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.941661 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:33.941888 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:33.941978 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:33.944100 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:33.944102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:33.944102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:33.944110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:33.945579 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.945900 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.945901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.945901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.945962 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:33.946131 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:33.946132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:33.946132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:33.946276 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:33.946343 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:33.946473 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 18:57:33.946550 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 18:57:33.946975 20222 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:57:33.947310 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.947208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38966)
0929 18:57:33.947255 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.947280 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:57:33.947631 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.947539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38978)
0929 18:57:33.947588 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.947611 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 18:57:33.947746 20222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:45569: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
I20260929 18:57:33.978894 20222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } attempt: 1
I20260929 18:57:33.979430 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.979292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38988)
0929 18:57:33.979361 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.979403 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:57:33.979835 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:33.979746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:38998)
0929 18:57:33.979795 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:33.979819 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 18:57:33.979965 20222 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:45569: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
W20260929 18:57:33.983001 20304 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:57:33.983203 20304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:57:33.983220 20304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:57:33.983242 20304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:57:33.984338 20304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:57:33.984372 20304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:57:33.984385 20304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:57:33.984397 20304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:57:33.984409 20304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:57:33.985787 20304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.71.148:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.71.190:40933,127.29.71.189:42229,127.29.71.188:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/krb5kdc/kudu_127.29.71.188.keytab
--principal=kudu/127.29.71.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.71.188:45569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.71.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 18:57:33.985980 20304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:57:33.986151 20304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:57:33.990983 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:33.991012 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:57:33.990998 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:57:33.991338 20304 server_base.cc:1034] running on GCE node
I20260929 18:57:33.991580 20304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:57:33.991860 20304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:57:33.993022 20304 hybrid_clock.cc:648] HybridClock initialized: now 1790708253993005 us; error 27 us; skew 500 ppm
Sep 29 18:57:33 dist-test-slave-q2kh krb5kdc[20141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:57:33.995594 20304 init.cc:377] Logged in from keytab as kudu/127.29.71.188@KRBTEST.COM (short username kudu)
I20260929 18:57:33.996039 20304 webserver.cc:460] Webserver started at http://127.29.71.188:44915/ using document root <none> and password file <none>
I20260929 18:57:33.996263 20304 fs_manager.cc:359] Metadata directory not provided
I20260929 18:57:33.996357 20304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:57:33.996517 20304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:57:33.997553 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data/instance:
uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "e2e323c767ed3089c94e9033c8814878"
server_key_iv: "b7d7e3e6269a6fe8210773a4bd7d36b3"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.997910 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/wal/instance:
uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "e2e323c767ed3089c94e9033c8814878"
server_key_iv: "b7d7e3e6269a6fe8210773a4bd7d36b3"
server_key_version: "encryptionkey@0"
I20260929 18:57:33.999390 20304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 18:57:34.000305 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:57:34.000494 20304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:57:34.000597 20304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data,/tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/wal
uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8"
format_stamp: "Formatted at 2026-09-29 18:57:33 on dist-test-slave-q2kh"
server_key: "e2e323c767ed3089c94e9033c8814878"
server_key_iv: "b7d7e3e6269a6fe8210773a4bd7d36b3"
server_key_version: "encryptionkey@0"
I20260929 18:57:34.000685 20304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:57:34.008802 20304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:57:34.011184 20304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:57:34.011355 20304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:57:34.011585 20297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } attempt: 2
I20260929 18:57:34.011999 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.011862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:39006)
0929 18:57:34.011923 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:34.011966 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 18:57:34.012358 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.012277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:39018)
0929 18:57:34.012318 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:34.012343 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 18:57:34.012466 20297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.71.188:45569: Network error: Client connection negotiation failed: client connection to 127.29.71.188:45569: connect: Connection refused (error 111)
I20260929 18:57:34.015522 20304 rpc_server.cc:307] RPC server started. Bound to: 127.29.71.188:45569
I20260929 18:57:34.015569 20371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.71.188:45569 every 8 connection(s)
I20260929 18:57:34.015940 20304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/data/info.pb
I20260929 18:57:34.016949 20372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:57:34.018568 20372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
I20260929 18:57:34.025389 29982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWZtMeW/build/release/bin/kudu as pid 20304
I20260929 18:57:34.025534 29982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWZtMeW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790707954762284-29982-0/minicluster-data/master-2/wal/instance
I20260929 18:57:34.025764 29982 external_mini_cluster.cc:1428] Setting key c8c909ed4dc71aa3e364ba19e2ab6252
I20260929 18:57:34.028407 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.019266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.29.71.190:40933)
0929 18:57:34.019325 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.019327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.019552 (+ 225us) server_negotiation.cc:407] Connection header received
0929 18:57:34.020052 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.020054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.020066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.020119 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.020756 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.021232 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.022305 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.022504 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.022592 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.024790 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.024791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.024791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.024799 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.027020 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.027476 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.027477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.027478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.027540 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.027775 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.027775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.027776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.027909 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.027975 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.028250 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.028342 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:57:34.028627 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.019240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.190:40933 (local address 127.0.0.1:36738)
0929 18:57:34.019435 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:34.019469 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:34.019941 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:34.020203 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:34.020206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:34.020457 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:34.020653 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.020657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.021341 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.021343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.022140 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.022142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.022616 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.022618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.022756 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.023276 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:34.023292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:34.024679 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:34.027109 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.027117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.027120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.027402 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.027610 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.027610 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.027611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.027708 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.027989 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:34.027993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:34.028113 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:57:34.028203 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:57:34.028400 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":110,"threads_started":1}
I20260929 18:57:34.029222 20372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:34.034797 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.026248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.29.71.188:45569)
0929 18:57:34.026417 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.026423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.026451 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:57:34.026474 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.026477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.026493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.026569 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.027231 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.028151 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.029181 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.029610 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.029679 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.032070 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.032080 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.032081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.032087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.033569 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.033974 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.033977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.033977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.034025 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.034242 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.034242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.034242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.034425 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.034576 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.034665 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.034725 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260929 18:57:34.036759 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.029479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.189:42229 (local address 127.0.0.1:35894)
0929 18:57:34.029530 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:34.029541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:34.029578 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:34.029737 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:34.029738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:34.029864 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:34.029904 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.029906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.030621 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.030622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.031092 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.031094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.031463 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.031464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.031536 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.031956 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:34.031959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:34.033317 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:34.035688 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.035688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.035689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.035902 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.036129 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.036130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.036131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.036185 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.036448 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:34.036449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:34.036489 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:57:34.036576 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:57:34.036665 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:57:34.036759 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.029567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.29.71.189:42229)
0929 18:57:34.029615 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.029618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.029625 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:57:34.029648 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.029649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.029660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.029714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.029981 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.030533 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.031180 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.031394 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.031467 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.033409 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.033411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.033411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.033418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.035603 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.035998 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.035999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.036000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.036055 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.036256 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.036257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.036257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.036372 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.036457 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.036603 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.036672 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 18:57:34.037176 20372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.190@KRBTEST.COM
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.188@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:57:34.043397 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.035824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.29.71.190:40933)
0929 18:57:34.035886 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.035888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.035895 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:57:34.035915 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.035916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.035925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.035995 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.036452 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.036995 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.037808 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.038065 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.038116 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.040533 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.040534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.040535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.040542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.042157 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.042509 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.042510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.042511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.042567 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.042794 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.042795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.042795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.042924 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.043086 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.043188 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.043270 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:57:34.043674 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.037423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:39048)
0929 18:57:34.037501 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:34.037511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:34.037554 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:34.037662 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:34.037663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:34.037825 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:34.037853 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.037855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.038620 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.038621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.039096 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.039097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.039482 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.039483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.039524 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.039910 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:34.039915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:34.041142 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:34.042643 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.042644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.042644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.042864 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.043044 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.043045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.043045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.043112 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.043383 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:34.043383 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:34.043415 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:57:34.043483 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:57:34.043567 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 18:57:34.043759 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.037471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.29.71.188:45569)
0929 18:57:34.037503 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.037505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.037540 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:57:34.037606 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.037607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.037616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.037662 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.037925 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.038556 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.039194 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.039422 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.039483 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.041225 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.041226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.041227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.041233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.042569 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.042928 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.042929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.042929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.042981 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.043174 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.043175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.043175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.043296 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.043361 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.043529 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.043624 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":15}
I20260929 18:57:34.045244 20372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: Bootstrap starting.
I20260929 18:57:34.046075 20372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:34.046427 20372 log.cc:824] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:34.047192 20372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: No bootstrap required, opened a new log
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.71.189@KRBTEST.COM
I20260929 18:57:34.049009 20372 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.049176 20372 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ecd0b6db9b4bcf8b666ceccfb3f5d8, State: Initialized, Role: FOLLOWER
I20260929 18:57:34.049273 20372 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.049594 20378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.049638 20372 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:34.049706 20378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:34.051227 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.043963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.29.71.189:42229)
0929 18:57:34.044001 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.044003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.044038 (+ 35us) server_negotiation.cc:407] Connection header received
0929 18:57:34.044110 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.044111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.044120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.044175 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.044692 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.045149 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.045867 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.046085 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.046200 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.048418 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.048419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.048420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.048427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.050102 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.050481 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.050482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.050483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.050541 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.050735 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.050736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.050736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.050854 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.051022 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.051059 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.051128 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 18:57:34.051699 20389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:34.051764 20389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:57:34.052548 29982 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:57:34.053363 20196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36742
I20260929 18:57:34.055116 20222 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } attempt: 2
I20260929 18:57:34.052224 20304 master_runner.cc:186] Error getting master registration for 127.29.71.190:40933: OK, instance_id { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" instance_seqno: 1790708253711265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:57:34.055258 20304 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.190@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:57:34.062621 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.055430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.29.71.188:45569)
0929 18:57:34.055482 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.055484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.055493 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:57:34.055532 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.055533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.055543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.055592 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.055925 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.056466 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.057434 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.057725 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.057790 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.060029 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.060030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.060031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.060038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.061420 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.061766 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.061768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.061768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.061826 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.062024 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.062025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.062025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.062215 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.062301 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.062448 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.062526 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:57:34.062621 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.055381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:39056)
0929 18:57:34.055430 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:34.055440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:34.055478 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:34.055617 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:34.055618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:34.055796 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:34.055847 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.055849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.056547 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.056548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.057358 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.057361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.057805 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.057806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.057867 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.058431 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:34.058437 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:34.059939 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:34.061522 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.061523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.061523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.061696 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.061903 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.061903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.061904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.061963 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.062291 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:34.062292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:34.062339 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 18:57:34.062425 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:57:34.062525 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 18:57:34.063875 20222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6: Bootstrap starting.
I20260929 18:57:34.064604 20222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:34.064910 20222 log.cc:824] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:34.065640 20222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6: No bootstrap required, opened a new log
I20260929 18:57:34.067070 20222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.067212 20222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9bc6a31ddf4e8daa396483900ccbe6, State: Initialized, Role: FOLLOWER
I20260929 18:57:34.067286 20222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.067581 20392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.067631 20222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:34.067672 20392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:34.067752 20196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:57:34.067823 20196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:34.068607 20196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.068856 20196 leader_election.cc:288] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [CANDIDATE]: Term 1 election: Requested vote from peers 54fdd297eddd45939bef0c88a60ef353 (127.29.71.189:42229), d5ecd0b6db9b4bcf8b666ceccfb3f5d8 (127.29.71.188:45569)
I20260929 18:57:34.069288 20346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8"
I20260929 18:57:34.069423 20346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:34.070400 20346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9bc6a31ddf4e8daa396483900ccbe6 in term 1.
I20260929 18:57:34.070050 20154 master_runner.cc:186] Error getting master registration for 127.29.71.189:42229: OK, instance_id { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" instance_seqno: 1790708253869218 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:57:34.072890 20154 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:57:34.073071 20171 leader_election.cc:302] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9bc6a31ddf4e8daa396483900ccbe6, d5ecd0b6db9b4bcf8b666ceccfb3f5d8; no voters:
I20260929 18:57:34.073185 20392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:57:34.073341 20392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [term 1 LEADER]: Becoming Leader. State: Replica: 0c9bc6a31ddf4e8daa396483900ccbe6, State: Running, Role: LEADER
I20260929 18:57:34.073441 20392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.073901 20392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c9bc6a31ddf4e8daa396483900ccbe6. Latest consensus state: current_term: 1 leader_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.074004 20392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:34.076766 20413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:57:34.077358 20413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:57:34.081662 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.072985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.29.71.190:40933)
0929 18:57:34.073028 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.073030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.073040 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:34.073062 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.073062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.073072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.073135 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.073971 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.074751 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.075796 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.076117 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.076175 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.078665 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.078666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.078667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.078674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.080287 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.080774 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.080776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.080776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.080832 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.081048 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.081048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.081049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.081173 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.081248 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.081484 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.081568 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 18:57:34.083344 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.074139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.29.71.189:42229)
0929 18:57:34.074199 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.074201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.074209 (+ 8us) server_negotiation.cc:407] Connection header received
0929 18:57:34.074232 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.074232 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.074241 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.074294 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.075020 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.075710 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.076676 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.076966 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.077036 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.079816 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.079817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.079818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.079825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.081435 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.081884 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.081885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.081886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.081958 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.082541 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.082542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.082543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.082688 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.082769 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.083163 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.083253 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:57:34.086771 20346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [term 1 FOLLOWER]: Refusing update from remote peer 0c9bc6a31ddf4e8daa396483900ccbe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:34.087159 20411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:34.088649 20378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c9bc6a31ddf4e8daa396483900ccbe6. Latest consensus state: current_term: 1 leader_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.088742 20378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:34.089820 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.071744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.29.71.188:45569)
0929 18:57:34.071782 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.071784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.082012 (+ 10228us) server_negotiation.cc:407] Connection header received
0929 18:57:34.082122 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.082123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.082134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.082185 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.082775 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.083482 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.084436 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.084639 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.084719 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.086069 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.086070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.086070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.086078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.088234 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.088648 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.088649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.088649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.088988 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.089231 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.089232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.089232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.089375 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.089448 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.089653 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.089730 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 18:57:34.090862 20411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.090950 20411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: This master's current role is: LEADER
I20260929 18:57:34.091209 20415 mvcc.cc:205] Tried to move back new op lower bound from 7334741008736145408 to 7334741008685629440. Current Snapshot: MvccSnapshot[applied={T|T < 7334741008736145408}]
I20260929 18:57:34.091430 20378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.091502 20378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:34.091952 20416 mvcc.cc:205] Tried to move back new op lower bound from 7334741008736145408 to 7334741008685629440. Current Snapshot: MvccSnapshot[applied={T|T < 7334741008736145408 or (T in {7334741008736145408})}]
I20260929 18:57:34.094574 20413 catalog_manager.cc:1347] Generated new cluster ID: dccb5acfbb504fdc8eeca08fc35d8501
I20260929 18:57:34.094633 20413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:57:34.103603 20297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } attempt: 3
Sep 29 18:57:34 dist-test-slave-q2kh krb5kdc[20141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.71.189@KRBTEST.COM for kudu/127.29.71.188@KRBTEST.COM
I20260929 18:57:34.111714 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.103893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.71.188:45569 (local address 127.0.0.1:39078)
0929 18:57:34.103975 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0929 18:57:34.103987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:57:34.104024 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:57:34.104286 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:57:34.104287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:57:34.104482 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:57:34.104531 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.104533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.105458 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.105458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.106294 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:57:34.106297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.106793 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.106794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:57:34.106859 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.107360 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:57:34.107366 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:57:34.108847 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:57:34.110694 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.110695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.110696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.110909 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.111102 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:57:34.111102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:57:34.111103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:57:34.111156 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:57:34.111418 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:57:34.111419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:57:34.111461 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 18:57:34.111521 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:57:34.111610 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 18:57:34.112049 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:34.104117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.29.71.188:45569)
0929 18:57:34.104157 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.104159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.104166 (+ 7us) server_negotiation.cc:407] Connection header received
0929 18:57:34.104187 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.104187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.104196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.104242 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.104613 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.105367 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.106395 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:34.106682 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:34.106740 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:34.108942 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:57:34.108943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:57:34.108943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:57:34.108950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:57:34.110605 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.110983 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.110985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.110985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.111038 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:57:34.111217 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:57:34.111217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:57:34.111218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:57:34.111335 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:57:34.111394 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:34.111887 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 18:57:34.111960 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 18:57:34.113538 20297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: Bootstrap starting.
I20260929 18:57:34.114302 20297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: Neither blocks nor log segments found. Creating new log.
I20260929 18:57:34.114604 20297 log.cc:824] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: Log is configured to *not* fsync() on all Append() calls
I20260929 18:57:34.115644 20297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: No bootstrap required, opened a new log
I20260929 18:57:34.117738 20297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.117894 20297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54fdd297eddd45939bef0c88a60ef353, State: Initialized, Role: FOLLOWER
I20260929 18:57:34.117975 20297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } }
I20260929 18:57:34.118363 20413 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:57:34.118443 20270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:57:34.119077 20413 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:57:34.119402 20270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [term 1 FOLLOWER]: Refusing update from remote peer 0c9bc6a31ddf4e8daa396483900ccbe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:57:34.119693 20411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:57:34.118469 20271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "54fdd297eddd45939bef0c88a60ef353"
I20260929 18:57:34.120693 20271 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9bc6a31ddf4e8daa396483900ccbe6 in term 1.
I20260929 18:57:34.118336 20297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:57:34.118326 20421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.123900 20421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:34.124008 20421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c9bc6a31ddf4e8daa396483900ccbe6. Latest consensus state: current_term: 1 leader_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.124114 20421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:34.124181 20421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.124267 20421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:57:34.127321 20423 mvcc.cc:205] Tried to move back new op lower bound from 7334741008852676608 to 7334741008685629440. Current Snapshot: MvccSnapshot[applied={T|T < 7334741008736145408}]
I20260929 18:57:34.127411 20413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6: Generated new TSK 0
I20260929 18:57:34.127586 20413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:57:34.128444 20392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9bc6a31ddf4e8daa396483900ccbe6" member_type: VOTER last_known_addr { host: "127.29.71.190" port: 40933 } } peers { permanent_uuid: "54fdd297eddd45939bef0c88a60ef353" member_type: VOTER last_known_addr { host: "127.29.71.189" port: 42229 } } peers { permanent_uuid: "d5ecd0b6db9b4bcf8b666ceccfb3f5d8" member_type: VOTER last_known_addr { host: "127.29.71.188" port: 45569 } } }
I20260929 18:57:34.128531 20392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c9bc6a31ddf4e8daa396483900ccbe6 [sys.catalog]: This master's current role is: LEADER
W20260929 18:57:34.128919 20436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:57:34.128980 20436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 18:57:34.154728 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.153072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.29.71.189:42229)
0929 18:57:34.153121 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.153123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.153996 (+ 873us) server_negotiation.cc:407] Connection header received
0929 18:57:34.154101 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.154103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.154115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.154147 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.154656 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35922: BlockingRecv error: recv got EOF from 127.0.0.1:35922 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 18:57:34.155493 20224 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.153497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.29.71.190:40933)
0929 18:57:34.153538 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.153540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.154980 (+ 1440us) server_negotiation.cc:407] Connection header received
0929 18:57:34.155075 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.155076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.155087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.155115 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.155411 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36754: BlockingRecv error: recv got EOF from 127.0.0.1:36754 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 18:57:34.156256 20376 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.153792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.29.71.188:45569)
0929 18:57:34.153837 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.153840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.155699 (+ 1859us) server_negotiation.cc:407] Connection header received
0929 18:57:34.155832 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.155833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.155843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.155872 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.156200 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39090: BlockingRecv error: recv got EOF from 127.0.0.1:39090 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 18:57:34.158663 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.157931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.29.71.189:42229)
0929 18:57:34.157964 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.157966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.158131 (+ 165us) server_negotiation.cc:407] Connection header received
0929 18:57:34.158278 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.158279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.158297 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.158326 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.158617 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35924: BlockingRecv error: recv got EOF from 127.0.0.1:35924 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 18:57:34.161564 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.160846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.29.71.189:42229)
0929 18:57:34.160877 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.160878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.161107 (+ 229us) server_negotiation.cc:407] Connection header received
0929 18:57:34.161199 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.161200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.161209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.161237 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.161521 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35936: BlockingRecv error: recv got EOF from 127.0.0.1:35936 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 18:57:34.165241 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.164517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.29.71.189:42229)
0929 18:57:34.164559 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.164561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.164787 (+ 226us) server_negotiation.cc:407] Connection header received
0929 18:57:34.164882 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.164883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.164892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.164920 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.165200 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35948: BlockingRecv error: recv got EOF from 127.0.0.1:35948 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 18:57:34.171146 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.170343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.29.71.189:42229)
0929 18:57:34.170381 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.170382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.170626 (+ 244us) server_negotiation.cc:407] Connection header received
0929 18:57:34.170721 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.170722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.170732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.170761 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.171099 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35956: BlockingRecv error: recv got EOF from 127.0.0.1:35956 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 18:57:34.181293 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.180325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.29.71.189:42229)
0929 18:57:34.180367 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.180369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.180674 (+ 305us) server_negotiation.cc:407] Connection header received
0929 18:57:34.180798 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.180799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.180812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.180844 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.181225 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35966: BlockingRecv error: recv got EOF from 127.0.0.1:35966 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 18:57:34.199918 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.198619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.29.71.189:42229)
0929 18:57:34.198779 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.198781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.199165 (+ 384us) server_negotiation.cc:407] Connection header received
0929 18:57:34.199320 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.199321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.199334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.199366 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.199856 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35976: BlockingRecv error: recv got EOF from 127.0.0.1:35976 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260929 18:57:34.234247 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.233240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.29.71.189:42229)
0929 18:57:34.233288 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.233290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.233583 (+ 293us) server_negotiation.cc:407] Connection header received
0929 18:57:34.233692 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.233694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.233706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.233737 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.234186 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35986: BlockingRecv error: recv got EOF from 127.0.0.1:35986 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 18:57:34.300300 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.299596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.29.71.189:42229)
0929 18:57:34.299662 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.299664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.299735 (+ 71us) server_negotiation.cc:407] Connection header received
0929 18:57:34.299781 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.299782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.299792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.299856 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.300219 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35998: BlockingRecv error: recv got EOF from 127.0.0.1:35998 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 18:57:34.430008 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.429398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.29.71.189:42229)
0929 18:57:34.429470 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.429472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.429499 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:57:34.429560 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.429561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.429571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.429622 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.429952 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36008: BlockingRecv error: recv got EOF from 127.0.0.1:36008 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 18:57:34.687965 20301 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:57:34.687315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.29.71.189:42229)
0929 18:57:34.687393 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:34.687396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:34.687405 (+ 9us) server_negotiation.cc:407] Connection header received
0929 18:57:34.687475 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:34.687476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:34.687486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:34.687528 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:57:34.687897 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36022: BlockingRecv error: recv got EOF from 127.0.0.1:36022 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:57:35.052244 20389 catalog_manager.cc:1259] Loaded cluster ID: dccb5acfbb504fdc8eeca08fc35d8501
I20260929 18:57:35.052356 20389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: loading cluster ID for follower catalog manager: success
I20260929 18:57:35.053267 20389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: acquiring CA information for follower catalog manager: success
I20260929 18:57:35.053589 20389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5ecd0b6db9b4bcf8b666ceccfb3f5d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:57:35.129392 20436 catalog_manager.cc:1259] Loaded cluster ID: dccb5acfbb504fdc8eeca08fc35d8501
I20260929 18:57:35.129465 20436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: loading cluster ID for follower catalog manager: success
I20260929 18:57:35.130340 20436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: acquiring CA information for follower catalog manager: success
I20260929 18:57:35.130671 20436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54fdd297eddd45939bef0c88a60ef353: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:57:35.204037 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:35.201182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.29.71.189:42229)
0929 18:57:35.201347 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:35.201350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:35.201360 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:35.201418 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:35.201419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:35.201467 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:35.201543 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:35.201741 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.202319 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.203198 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.203413 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.203459 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:35.203592 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:35.203700 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:35.203755 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:35.203915 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 18:57:35.203933 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260929 18:57:35.208519 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:35.205945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.29.71.188:45569)
0929 18:57:35.206039 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:35.206040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:35.206089 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:57:35.206208 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:35.206211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:35.206252 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:35.206330 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:35.206407 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.206873 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.207698 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.208017 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.208068 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:35.208197 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:35.208290 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:35.208348 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:35.208443 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 18:57:35.208459 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260929 18:57:35.213264 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:35.210259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.29.71.189:42229)
0929 18:57:35.210403 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:35.210405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:35.210415 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:57:35.210516 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:35.210517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:35.210529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:35.210610 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:35.210759 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.211623 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.212514 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.212740 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.212831 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:35.212999 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:35.213033 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:35.213090 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:35.213138 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 18:57:35.213159 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260929 18:57:35.213481 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:35.210313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.29.71.190:40933)
0929 18:57:35.210403 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:35.210405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:35.210590 (+ 185us) server_negotiation.cc:407] Connection header received
0929 18:57:35.210706 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:35.210707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:35.210758 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:35.210833 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:35.211063 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.211556 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.212484 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.212804 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.212843 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:35.213088 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:35.213201 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:35.213265 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:35.213377 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 18:57:35.213406 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":49,"threads_started":1}
I20260929 18:57:35.213635 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:57:35.210714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.29.71.188:45569)
0929 18:57:35.210756 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 18:57:35.210758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 18:57:35.210863 (+ 105us) server_negotiation.cc:407] Connection header received
0929 18:57:35.210911 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:57:35.210912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:57:35.210923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:57:35.210981 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:57:35.211107 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.211862 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.212844 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:57:35.213138 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:57:35.213174 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:57:35.213388 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:57:35.213422 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:57:35.213448 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0929 18:57:35.213541 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 18:57:35.213560 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 18:57:35.223850 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 20154
I20260929 18:57:35.229470 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 20229
I20260929 18:57:35.234670 29982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWZtMeW/build/release/bin/kudu with pid 20304
2026-09-29T18:57:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2801 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5602 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (300479 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS